design myami-python3 autoinstaller for leginon

[leginonbot]

---

Author Name: Anchi Cheng (@anchi2c)
Original Redmine Issue: 13359, https://emg.nysbc.org/redmine/issues/13359
Original Date: 2022-08-26
Original Assignee: Sargis Dallakyan

---

Needs an autoinstaller to help new user to have an all-in-one installation quickly. Please consider a modern option if the old way feels obsolete. This feature can end at a functional leginon with simulator. Once the form of this is determined from this, we can then add appion support.

Possible python options uses conda, pip/pyproject.toml, instead of setuptools so dependency can be included.

Gabe Lander uses, for example, conda, for his newer development of appion that includes leginon requirement.

conda create -n [envName] python==3.9.2 numpy==1.22.3 scipy==1.8.0 Pillow==9.1.0 joblib==1.1.0 matplotlib==3.5.1 PyMySQL==1.0.2 slackclient==2.9.3 wxPython==4.1.1 future==0.18.2 tqdm==4.64.0 psutil=5.9.1 pandas=1.4.3 scikit-image=0.19.3

I do similar with pip.

[leginonbot]

---

Original Redmine Comment
Author Name: Sargis Dallakyan (@dallakyan)
Original Date: 2022-08-28T18:43:12Z

---

Thank you for creating this issue Anchi. The old autoinstaller is working fine and I think it is not obsolete. Users who want to use conda or pip venv, can do that, then activate venv and run the autoinstaller. The autoinstaller also installs MySQL server that we can't do with conda or pip/pyproject.toml.

I've made changes in autoinstaller in python3 branch to make it work with python3. I also changed the part when it does pip install. It now downloads requirements.txt and does pip3 install -r ./requirements.txt.

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-08-30T22:02:27Z

---

Thanks. I tried it in a docker image of centos:7

I did first

yum install python3

and then ran the installer with

python3 centos7AutoInstallation.py

Here is what I've got, after changing in the script for it to git clone myami-python3 instead of trunk.

locate mpirun

Please wait......(This may take a few minutes.)

b''
locate: can not stat () `/var/lib/mlocate/mlocate.db': No such file or directory
Traceback (most recent call last):
  File "centos7AutoInstallation.py", line 1479, in <module>
    a.run()
  File "centos7AutoInstallation.py", line 1425, in run
    result = self.setupProcessServer()
  File "centos7AutoInstallation.py", line 409, in setupProcessServer
    self.processServerPackageEnable()
  File "centos7AutoInstallation.py", line 287, in processServerPackageEnable
    self.linkMpiRun()
  File "centos7AutoInstallation.py", line 262, in linkMpiRun
    if ( filename not in resultstring ):
TypeError: a bytes-like object is required, not 'str'

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-08-30T22:16:20Z

---

The install log is also full of errors. See attached.

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-08-30T22:18:01Z

---

By the way, the python install through yum is 3.6.8. Not sure if it will work. We have been developed in 3.8+

[leginonbot]

---

Original Redmine Comment
Author Name: Sargis Dallakyan (@dallakyan)
Original Date: 2022-08-31T01:59:34Z

---

Thank you Anchi. I fixed the issue with linkMpiRun. I'm using Python 3.9.5 installed locally.

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-08-31T03:38:58Z

---

How should I (and to instruct user to) install 3.9 on CentOS7 for autoinstallor to run ?

[leginonbot]

---

Original Redmine Comment
Author Name: Sargis Dallakyan (@dallakyan)
Original Date: 2022-08-31T04:17:20Z

---

I checked online; to install 3.9 on CentOS7 currently, we or users need to install it from the source:
https://computingforgeeks.com/install-latest-python-on-centos-linux/
https://www.inmotionhosting.com/support/server/linux/install-python-3-9-centos-7/

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-08-31T21:05:45Z

---

There are so many steps to do on the build of python 3.9 as well as linking it so that autoinstallor will work. In the spirit of simplify this for users to quickly set up, please put these build instruction together to a shell script so that it can be run before centos7AutoInstallation.py and work correctly. Also please add your final instruction to wiki that will replace this part in [[Autoinstaller_for_CentOS]]. Mark it as myami-python3 so it can be found later. Thanks.

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-08-31T21:11:53Z

---

I also got this warning since this installation is done as root. Is this a concern ?

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-08-31T21:33:49Z

---

ln -s /usr/local/bin/python3.9 python3
python3 centos7AutoInstallation.py

results in packages still installed in python2.7 site-packages.

python2
Python 2.7.5 (default, Jun 28 2022, 15:30:04) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyscope
>>> 

sh-4.2# python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyscope
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pyscope'

If I replace the default python with

ln -s /usr/local/bin/python3.9 python

then I get many python2 syntax error like this.

yum -y install yum-fastestmirror.noarch yum-utils.noarch

Please wait......(This may take a few minutes.)

b''
  File "/usr/bin/yum", line 30
    except KeyboardInterrupt, e:
                            ^
SyntaxError: invalid syntax

I don't think this approach works.

[leginonbot]

---

Original Redmine Comment
Author Name: Sargis Dallakyan (@dallakyan)
Original Date: 2022-09-02T03:07:02Z

---

I've made new commits to convert all python to python3. Please unlink python from /usr/local/bin/python3.9. The yum error is because it needs python2. Let's assume python is python2 as is it the default in CentOS7.

Let's see if it works with the default python36 that comes with CentOS7. If there are no new features that require python3.9, we don't need instructions to install python3.9. Thank you.

[leginonbot]

---

Original Redmine Comment
Author Name: Anchi Cheng (@anchi2c)
Original Date: 2022-09-02T03:54:49Z

---

Please do the testing yourself first to completion. I've already tried linking python to python3. I can't accept python 3.6 solution. There are features needing at least python 3.8.

I haven't thought of this, I wonder if it will help if we use #!/usr/bin/env python3 instead of #!/usr/bin/env python in our script.

Anyway, please run your test to completion before asking me to test again. Thank you.

[leginonbot]

---

Original Redmine Comment
Author Name: Sargis Dallakyan (@dallakyan)
Original Date: 2022-09-03T19:02:31Z

---

Thank you Anchi. So the prerequisite is to have Python 3.8 or 3.9 installed. We can have instructions or a script to do this.

For now, with python 3.9 installed in /usr/local (which python3 > /usr/local/bin/python3) the autoinstall works using @python3 centos7AutoInstallation.py@

There is an issue with setting up mysql password. I had to reset mysql root password using https://www.digitalocean.com/community/tutorials/how-to-reset-your-mysql-or-mariadb-root-password. This is a typical post-install issue with securing the database that is also present in python2 version.

Now with mysql root password set, the installation completes without error, and all required packages are installed. The start-leginon.py has the right python interpreter set (#!/usr/local/bin/python3). However, I can't start leginon:

1. start-leginon.py
   Error: (2003, "Can't connect to MySQL server on 'localhost' ([Errno 111] Connection refused)")

I'm able to connect using mysql -u root -h localhost -p. I'll investigate why. Please let me know if you can reproduce this or have any insights on how to fix this. Thank you.

[leginonbot]

---

Original Redmine Comment
Author Name: Sargis Dallakyan (@dallakyan)
Original Date: 2022-09-03T23:04:20Z

---

The problem with 'Can't connect to MySQL server' has been resolved. I had to restart mariadb to make it work. Now, I'm getting import errors. Are you able to start leginon with Python 3? I can go over and fix all remaining import errors if needed.