Update bashrc and postgres-connection roles #610

sanjaysrikakulam · 2023-02-15T12:11:21Z

Updates:

This update will now allow us to add the bashrc content to more than 1 user. However, the default behaviour is retained (for backwards compatibility) i.e., by default it adds the contents to the galaxy user. Please make sure that the user (and the home directory) already exists.

Example:

  roles:
    - role: usegalaxy-eu.bashrc
      vars:
        bashrc_users:
          - uname: galaxy
            uhome: /home/galaxy
            gname: galaxy

          - uname: stats
            uhome: /home/stats
            gname: stats
    - role: hxr.postgres-connection
      vars:
        pgc_users:
          - uname: galaxy
            uhome: /home/galaxy
            gname: galaxy
            pguser: "{{ your_pg_user }}"
            pgpass: "{{ your_pg_password }}"
            pgdatabase: your_pg_database

          - uname: stats
            uhome: /home/stats
            gname: stats
            pguser: "{{ your_pg_user }}"
            pgpass: "{{ your_pg_password }}"
            pgdatabase: your_pg_database

Added ENVs required for certain gxadmin commands
Added a new change_to_jwd function that uses the new /usr/local/bin/galaxy_jwd python script. This script and the function accept only the job id. The new alias is chg2jwd and also the old method chg2wd is available just in case.
Added galaxy read-only DB user creds to the vault

The script /usr/local/bin/galaxy_jwd needs the following ENVs and a ~/.pgpass file to work
ENVs:

GALAXY_CONFIG_FILE: Path to the galaxy.yml file
GALAXY_LOG_DIR: Path to the Galaxy log directory
PGDATABASE: Name of the Galaxy database
PGUSER: Galaxy database user
PGHOST: Galaxy database host

~/.pgpass format: <pg_host>:5432:*:<pg_user>:<pg_password>

Additionally, the hxr.postgres-connection role has been updated to add the required PG configuration to the bashrc file of any user (as long as the user is already present and the home directory is created). The update is backwards compatible (default vars are set accordingly to make sure the role retains its default behaviour).

Updates: 1. This update will now allow us to add the bashrc content to more than 1 user. However, the default behaviour is retained (for backwards compatibility) i.e., by default it adds the contents to the galaxy user. 2. Added ENVs required for certain gxadmin commands 3. Added new change_to_jwd function that uses the new get_jwd bash script. This script and the function accepts only the job id. The new alias is chg2jwd and also the old method chg2wd is available just in case.

bgruening · 2023-02-15T23:06:13Z

roles/usegalaxy-eu.bashrc/templates/get_jwd.sh.j2

@@ -0,0 +1,61 @@
+#!/bin/bash
+# Description: Get the job working directory using the job id and the object store configuration XML file


You have a python script for that, isn't it? How does this script relate to it?

I have a Python script for cleaning up JWD's. I can also tweak that script and this functionality to it, and we can call a sub-command get_jwd via the command line that would return the JWD path and another sub-command cleanup_jwd that would clean up the JWDs if you would like.

I think it would make sense to have this logic only in one place and not in two different scripts. Especially if you are planning to upstream your python script, we could reuse it easily.

I have totally updated the script. Here is the gist. Please have a look and share your review.

@sanjaysrikakulam this looks largely ok, just a few small things. I guess its ready for trying to upstream it.

@bgruening Created PR: galaxyproject/galaxy#15618

Shall I remove this bash script and replace it with the python script (in the above PR) until the upstream merge it and release it?

Sounds good!

I have updated this PR. Please review everything and let me know if any changes are required.

Remove the bash script template Replace with the new python script Add PGHOST ENV to the bashrc

Now we can use the variables to add the configuration to any system user and use any galaxy database credentials

mira-miracoli

As I understand we have to invoke the bashrc role for every user. Would it be possible to use a list instead?
Otherwise it looks good to me.

mira-miracoli · 2023-02-21T08:35:03Z

roles/usegalaxy-eu.bashrc/defaults/main.yml

@@ -0,0 +1,4 @@
+---
+user_name: "{{ galaxy_user.name }}"


I think its good to use more specific variable names, to avoid unintended overwriting of other variables and less confusion in the vars files. e.g.

Suggested change

user_name: "{{ galaxy_user.name }}"

bashrc_user_name: "{{ galaxy_user.name }}"

Ansible does not support looping over a block

All tasks in a block should be moved out to another tasks file and then should be included in the main.yml where we can loop. I can implement this since you don't like the idea of invoking the role for every user.

I like that more, because the variables can stay in a vars file and are not in the playbook. Thank you!

Here is the update 3b6594f

I have also updated the example above

This change will allow to include role once instead of including/invoking it for ever user. The required variables can now be defined as a list and if the users exist the required tasks will be applied.

mira-miracoli

Thank you @sanjaysrikakulam

Look at this PR for changes: usegalaxy-eu#610

bgruening · 2023-02-21T18:59:15Z

roles/usegalaxy-eu.bashrc/tasks/bashrc_tasks.yml

+          alias glg='journalctl -fu galaxy-gunicorn@* | grep -v -e "/api/upload/hooks" -e "/history/current_history_json"'
+          alias glh='journalctl -f -u galaxy-handler@*'
+          alias glw='journalctl -f -u galaxy-workflow-scheduler@*'
+          alias glc='journalctl -fu galaxy-celery@*'


celery is not on the head nodes anymore, or is this role also used on the celery host?

No, the role is not used in the celery.yml playbook. I think it's legacy code. I will remove it and add a new commit.

Updated: 229c0cb

Refer: usegalaxy-eu#610 (comment)

sanjaysrikakulam added 2 commits February 15, 2023 12:38

fix new line issue

3210570

sanjaysrikakulam requested review from bgruening and mira-miracoli February 15, 2023 12:11

bgruening reviewed Feb 15, 2023

View reviewed changes

sanjaysrikakulam mentioned this pull request Feb 16, 2023

WIP: Add maintenance node playbook and vars file #614

Draft

sanjaysrikakulam added 2 commits February 20, 2023 14:01

Replace the bash script with the python script

44ca6da

Remove the bash script template Replace with the new python script Add PGHOST ENV to the bashrc

Update role to allow adding config to any sys user

ac04158

Now we can use the variables to add the configuration to any system user and use any galaxy database credentials

mira-miracoli requested changes Feb 21, 2023

View reviewed changes

Update bashrc and postgres-connection roles

3b6594f

This change will allow to include role once instead of including/invoking it for ever user. The required variables can now be defined as a list and if the users exist the required tasks will be applied.

mira-miracoli approved these changes Feb 21, 2023

View reviewed changes

sanjaysrikakulam added a commit to sanjaysrikakulam/usegalaxy-eu_infrastructure-playbook that referenced this pull request Feb 21, 2023

Update bashrc and postgres-connection role's vars

0ad278c

Look at this PR for changes: usegalaxy-eu#610

sanjaysrikakulam changed the title ~~Update bashrc~~ Update bashrc and postgres-connection roles Feb 21, 2023

bgruening reviewed Feb 21, 2023

View reviewed changes

Remove celery alias

229c0cb

Refer: usegalaxy-eu#610 (comment)

bgruening approved these changes Feb 22, 2023

View reviewed changes

sanjaysrikakulam merged commit 39fc58b into usegalaxy-eu:master Mar 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update bashrc and postgres-connection roles #610

Update bashrc and postgres-connection roles #610

sanjaysrikakulam commented Feb 15, 2023 •

edited

Loading

bgruening Feb 15, 2023

sanjaysrikakulam Feb 16, 2023

bgruening Feb 16, 2023

sanjaysrikakulam Feb 16, 2023

bgruening Feb 16, 2023

sanjaysrikakulam Feb 20, 2023

sanjaysrikakulam Feb 20, 2023

bgruening Feb 20, 2023

sanjaysrikakulam Feb 20, 2023

mira-miracoli left a comment

mira-miracoli Feb 21, 2023

sanjaysrikakulam Feb 21, 2023 •

edited

Loading

sanjaysrikakulam Feb 21, 2023

mira-miracoli Feb 21, 2023

sanjaysrikakulam Feb 21, 2023 •

edited

Loading

mira-miracoli left a comment

bgruening Feb 21, 2023

sanjaysrikakulam Feb 22, 2023 •

edited

Loading

sanjaysrikakulam Feb 22, 2023

		@@ -0,0 +1,61 @@
		#!/bin/bash
		# Description: Get the job working directory using the job id and the object store configuration XML file

	user_name: "{{ galaxy_user.name }}"
	bashrc_user_name: "{{ galaxy_user.name }}"

Update bashrc and postgres-connection roles #610

Update bashrc and postgres-connection roles #610

Conversation

sanjaysrikakulam commented Feb 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mira-miracoli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam Feb 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam Feb 21, 2023 • edited Loading

Choose a reason for hiding this comment

mira-miracoli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam Feb 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanjaysrikakulam commented Feb 15, 2023 •

edited

Loading

sanjaysrikakulam Feb 21, 2023 •

edited

Loading

sanjaysrikakulam Feb 21, 2023 •

edited

Loading

sanjaysrikakulam Feb 22, 2023 •

edited

Loading