Difference between revisions of "Scratch pad"

From SPOT CHECK IT
Jump to: navigation, search
 
(2 intermediate revisions by the same user not shown)
Line 210: Line 210:


Hamming distance js library find dictionary matches and misspelled words  
Hamming distance js library find dictionary matches and misspelled words  
NLP library to get nouns and send to the synonymy library to expand search 
https://github.com/spencermountain/compromise 


'''CURRENT STATUS:'''
'''CURRENT STATUS:'''


Mediawiki search PHP  
Mediawiki search PHP  
     4 - PHP Xbert ES system (Not implemented yet) (Make xbt node trees for scripting )       
     3 - PHP Xbert ES system (Not implemented yet) (Make xbt node trees for scripting )       
       - JS ES search (preprocessor)
       - JS ES search (preprocessor)
         1 -  Dict match and mispelled words (Working with now)
         * typo.js Dict match and misspelled words  
         2 - dictionary.js This will expand the searched word list to include synonymys
         1 -  compromise.js : NLP search for nouns and send to synonymy to expand search. 
     3 - Audio Voice recog Input
        1 - dictionary.js This will expand the searched word list to include synonymy
     2 - Audio Voice recog Input

Latest revision as of 22:27, 14 May 2022

This page is to keep items I will organize on the page later or ideas.

Analyse and calculate php-fpm runner settings https://megamorf.gitlab.io/2021/03/08/analyse-and-calculate-php-fpm-runner-settings/


Get average memory of the php-fpm process

ps --no-headers -o "rss,cmd" -C php-fpm | awk '{ sum+=$1 } END { printf ("%d%s\n", sum/NR/1024,"M") }'


Get memory of all php-fpm processes

ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | grep php-fpm


How many cores does server have.

echo Cores = $(( $(lscpu | awk '/^Socket/{ print $2 }') * $(lscpu | awk '/^Core/{ print $4 }') ))


php-fpm settings

https://thisinterestsme.com/php-fpm-settings/


Top 10 processes using memory

ps auxw|head -1;ps auxw|sort -rn -k4|head -10


Get 20 Highest mem users.

echo [PID]  [MEM]  [PATH] &&  ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20

Get highest cpu users.

ps -eo pcpu,pid,user,args | sort -k 1 -r | head -20


Apache tuning.

https://hostadvice.com/how-to/how-to-tune-and-optimize-performance-of-apache/

Estimate the Thrashing Point

https://www.liquidweb.com/kb/apache-performance-tuning-swap-memory/


ps aux | grep 'httpd' | awk '{print $6/1024;}' | awk '{sum += $1;} END {print sum " MB total mem usage";}'

ps aux | grep 'httpd' | awk '{print $6/1024;}' | awk '{avg += ($1 - avg) / NR;} END {print avg " MB avg mem usage";}'


Useful shell script commands.

https://www.lancaster.ac.uk/staff/tipper/writing/luui/html/unix-intro012.html


https://www.redhat.com/sysadmin/one-line-linux-commands


Make multiple directories in one command.

mkdir -p -v /home/josevnz/tmp/{dir1,anotherdir,similardir}


Make backup via encrypted transfer:

tar --create --directory /home/josevnz/tmp/ --file - *| \

ssh raspberrypi "tar --directory /home/josevnz \

--verbose --list --file -"


Find 10 largest directories.

du -sk /var/log/* | sort -r -n | head -10


Colorize Grep

grep -i --color=auto


https://linuxcommandlibrary.com/basic/oneliners

Remove all files that dont match ext.

rm !(*.foo|*.bar|*.baz)

vi a remote file.

vi scp://username@host//path/to/somefile


Clear line in terminal

CTRL+u


so searching for "data scale" will find any results with data or with scale, "+data +scale" will only find results with both data and scale

and, thank you ignored.

stopword file list to add to ignore list.

If the innodb_ft_user_stopword_table option is set, the stopwords are taken from that table. Else, if the innodb_ft_server_stopword_table option is set, the stopwords are taken from that table. Otherwise, a built-in set of default stopwords is used."

Mediawiki rebuild index stopwords.

Types of Full-Text search IN NATURAL LANGUAGE MODE¶

https://mariadb.com/kb/en/full-text-index-overview/#in-boolean-mode


https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/MWSearch/+/bec5950ac833dd9d808fd0f0a2de0d58e36a3e06/MWSearch_body.php


http://sphinxsearch.com/blog/2020/07/06/sphinx-3-3-1-released/

Add voice speech to text with php to javascript.

https://makitweb.com/how-to-add-speech-recognition-to-the-website-javascript/

Expert System search for using bash commands.

http://law-expert.sourceforge.net/

Xbert program. Has xbt and know file format for guiding search or clicks to a narrow solution.

https://www.phpclasses.org/package/10346-PHP-Ask-questions-and-make-decisions-based-on-answers.html

Maybe make option to use online Thesarus with PHP for search

https://words.bighugelabs.com/site/api

Use Synonym in natural speech search to increase page resource finds.

composer require hugsbrugs/php-synonym

https://github.com/hugsbrugs/php-synonym

Check to see if site has http2 enabled.

curl --http2 domain.tld


Stack track all of the php-fpm processes.

strace -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')


Stack Trace Apache process.

ps h --ppid $(cat /var/run/apache2/httpd.pid) | awk '{print"-p " $1}' | xargs strace


Stack Trace a process and threads

strace -s 80 -fp 46339


Use strace to track process network events.

strace -e trace=%network -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')

Use Strace to track all system calls.

strace -e trace=connect -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')

Use strace to track all file operations

strace -s 1024 -e trace=%desc -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')

Use strace to track all process operations

strace -s 1024 -e trace=%process -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')

Use strace to print paths with file descriptors

strace -s 1024 -y -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')

Use strace to print socket file descriptors.

strace -s 1024 -yy -e trace=%network -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')


https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition/SpeechRecognition

https://coqui.ai/models

https://www.bashoneliners.com/oneliners/newest/

https://wiki.mozilla.org/Web_Speech_API_-_Speech_Recognition

Watch website for header access failure. Returns 200 if good.

curl -s -I -o /dev/null -w '%{http_code}' https://domain.tld

Make search more robust, if someone searches for "alternate" it will also tag "backup" in the search

https://github.com/FinNLP/synonyms

library is dictionary.js


Hamming distance js library find dictionary matches and misspelled words

NLP library to get nouns and send to the synonymy library to expand search

https://github.com/spencermountain/compromise

CURRENT STATUS:

Mediawiki search PHP

   3 - PHP Xbert ES system (Not implemented yet) (Make xbt node trees for scripting )       
     - JS ES search (preprocessor)
       * -  typo.js Dict match and misspelled words 
       1 -  compromise.js : NLP search for nouns and send to synonymy to expand search.   
       1 - dictionary.js This will expand the searched word list to include synonymy 
   2 - Audio Voice recog Input