New production crawler

[michielbdejong]

We've set up the server at 206.81.0.208 with a user 'runner' that has both @madoleary's and my ssh key authorized.
It can run cd ~/server ; npx ota track in a screen.
I pull the data to my laptop and then relay it to github:

git clone [email protected]:tosdr/tosdr-versions
cd tosdr-versions
git remote add server runner@ota-tosdr-8gb:engine/data/versions
git fetch server
git merge server/main
git push

And same for tosdr-snapshots.

[michielbdejong]

I'll create a 4Gb one as the production one, to save us money

[michielbdejong]

Pointed ota.tosdr.org to 159.223.154.84 in DNS

[michielbdejong]

Copied most updated instructions from #1160 :
So, I think the instructions that work are:

• ssh into a fresh Ubuntu 20.04 box. Then:

adduser crawler

(pick a good hard-to-guess password that will be used for sudo later)
(leave room number etc empty)

usermod -aG sudo crawler
mkdir /home/crawler/.ssh
cp ~/.ssh/authorized_keys /home/crawler/.ssh
chown -R crawler /home/crawler/.ssh
su crawler
cd ~
sudo ls
wget https://raw.githubusercontent.com/tosdr/ota-engine/main/ota-tosdr-server-init.sh
time sh ./ota-tosdr-server.sh
cd engine
source ~/.bashrc
nvm install 20
nvm use 20
npm install
npx ota track --services "Musi"

[michielbdejong]

I accidentally picked Ubuntu 24.04 and confirmed that this way of installing Puppeteer on a server really doesn't work for that OS version. Rebuilding it as Ubuntu 20.04 now.

[michielbdejong]

sudo apt install certbot
sudo certbot certonly --standalone

[michielbdejong]

Running

NODE_OPTIONS=--max_old_space_size=8000 npx ota track

in a screen now, let's see how it does on a 4Gb server.

[michielbdejong]

The run took 35 minutes, so that's great! Will add the github user and schedule it hourly.

[michielbdejong]

Cron job is running now

crawler@ota:~$ tail -f Wed\ Sep\ 11\ 13\:55\:01\ UTC\ 2024.log 

I set up a git ssh key, will see if the engine will do a git push. if not, we can add it to the /home/crawler/hourly.sh script

[michielbdejong]

The robustness of the cronjob is still not acceptable for production usage, so this work is now blocked on #1174

[michielbdejong]

Done