Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update bashrc and postgres-connection roles #610

Merged
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions roles/usegalaxy-eu.bashrc/defaults/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
user_name: "{{ galaxy_user.name }}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think its good to use more specific variable names, to avoid unintended overwriting of other variables and less confusion in the vars files. e.g.

Suggested change
user_name: "{{ galaxy_user.name }}"
bashrc_user_name: "{{ galaxy_user.name }}"

Copy link
Member Author

@sanjaysrikakulam sanjaysrikakulam Feb 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ansible does not support looping over a block

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All tasks in a block should be moved out to another tasks file and then should be included in the main.yml where we can loop. I can implement this since you don't like the idea of invoking the role for every user.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that more, because the variables can stay in a vars file and are not in the playbook. Thank you!

Copy link
Member Author

@sanjaysrikakulam sanjaysrikakulam Feb 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the update 3b6594f

I have also updated the example above

user_home: "{{ galaxy_user.home }}"
group_name: "{{ galaxy_group.name }}"
object_store_config_file: "{{ galaxy_config_dir }}/object_store_conf.xml"
190 changes: 117 additions & 73 deletions roles/usegalaxy-eu.bashrc/tasks/main.yml
Original file line number Diff line number Diff line change
@@ -1,85 +1,129 @@
---
- name: Check for bashrc
stat:
path: "{{ galaxy_user.home }}/.bashrc"
register: bashrc_stat_out
- name: Get all users
getent:
database: passwd
split: ':'

- name: Copy default bashrc when not existing
copy:
src: /etc/skel/.bashrc
dest: "{{ galaxy_user.home }}/.bashrc"
remote_src: yes
mode: 0640
owner: "{{ galaxy_user.name }}"
group: "{{ galaxy_group.name | default(galaxy_group) }}"
when: not bashrc_stat_out.stat.exists
- name: Debug info when user does not exist
debug:
msg: "INFO: User {{ user_name }} does not exist"
when: (not user_name in getent_passwd.keys())

- name: Check for bashprofile
stat:
path: "{{ galaxy_user.home }}/.bash_profile"
register: bashprofile_stat_out
- name: Check and add/update bashrc when user user exists
block:
- name: Check for bashrc
stat:
path: "{{ user_home }}/.bashrc"
register: bashrc_stat_out

- name: Check for profile
stat:
path: "{{ galaxy_user.home }}/.profile"
register: profile_stat_out
- name: Copy default bashrc when not existing
copy:
src: /etc/skel/.bashrc
dest: "{{ user_home }}/.bashrc"
remote_src: yes
mode: 0640
owner: "{{ user_name }}"
group: "{{ group_name }}"
when: not bashrc_stat_out.stat.exists

- name: Copy default bashprofile when not existing
copy:
src: /etc/skel/.bash_profile
dest: "{{ galaxy_user.home }}/.bash_profile"
remote_src: yes
mode: 0640
owner: "{{ galaxy_user.name }}"
group: "{{ galaxy_group.name | default(galaxy_group) }}"
when: not bashprofile_stat_out.stat.exists and not profile_stat_out.stat.exists
- name: Check for bashprofile
stat:
path: "{{ user_home }}/.bash_profile"
register: bashprofile_stat_out

- name: Insert some aliases
blockinfile:
path: "{{ galaxy_user.home }}/.bashrc"
marker: "# {mark} ANSIBLE MANAGED BLOCK"
content: |
# User specific aliases and functions
function change_to_wd() {
USAGE="Please provide a Galaxy job ID or a Condor job ID"
if (( $# == 0 )); then
- name: Check for profile
stat:
path: "{{ user_home }}/.profile"
register: profile_stat_out

- name: Copy default bashprofile when not existing
copy:
src: /etc/skel/.bash_profile
dest: "{{ user_home }}/.bash_profile"
remote_src: yes
mode: 0640
owner: "{{ user_name }}"
group: "{{ group_name }}"
when: not bashprofile_stat_out.stat.exists and not profile_stat_out.stat.exists

- name: Copy template get_jwd.sh bash script
template:
src: get_jwd.sh.j2
dest: /usr/local/bin/get_jwd
owner: galaxy
group: galaxy
mode: 0755

- name: Insert some aliases and functions
blockinfile:
path: "{{ user_home }}/.bashrc"
marker: "# {mark} ANSIBLE MANAGED BLOCK"
content: |
# User specific aliases and functions
function change_to_wd() {
USAGE="Please provide a Galaxy job ID or a Condor job ID"
if (( $# == 0 )); then
echo $USAGE
return 0;
fi
for i in "$@"; do
if [[ "$i" = --help || "$i" = -h ]]; then
echo $USAGE
return 0;
fi
done
JID=$1
WD=$(dirname `condor_q -autoformat Cmd ClusterId | grep ${JID} | cut -f1 -d' '` || dirname `condor_history -autoformat Cmd ClusterId | grep ${JID} | cut -f1 -d' '` || find "{{ galaxy_config['galaxy']['job_working_directory'] }}""/0"${JID:0:2}"/"${JID:2:3} -maxdepth 1 -type d -name ${JID})
cd $WD
}

# Uses the /usr/local/bin/get_jwd bash script to change to the job working directory
function change_to_jwd() {
USAGE="Please provide a Galaxy job ID"
if (( $# == 0 )); then
echo $USAGE
return 0;
fi
for i in "$@"; do
if [[ "$i" = --help || "$i" = -h ]]; then
echo $USAGE
return 0;
fi
done
JID=$1
WD=$(dirname `condor_q -autoformat Cmd ClusterId | grep ${JID} | cut -f1 -d' '` || dirname `condor_history -autoformat Cmd ClusterId | grep ${JID} | cut -f1 -d' '` || find "{{ galaxy_config['galaxy']['job_working_directory'] }}""/0"${JID:0:2}"/"${JID:2:3} -maxdepth 1 -type d -name ${JID})
cd $WD
}
fi

JID=$1
JWD=$(/usr/local/bin/get_jwd $JID)
cd $JWD
}

alias gl='journalctl -f -u galaxy-*'
alias glg='journalctl -fu galaxy-gunicorn@* | grep -v -e "/api/upload/hooks" -e "/history/current_history_json"'
alias glh='journalctl -f -u galaxy-handler@*'
alias glw='journalctl -f -u galaxy-workflow-scheduler@*'
alias glc='journalctl -fu galaxy-celery@*'
alias cu='journalctl -u galaxy-gunicorn@*.service --since "10 minutes ago" | grep "/history/current_history_json" | awk "{print \$11}" | sort -u | wc -l'
alias chg2wd='change_to_wd'
alias gl='journalctl -f -u galaxy-*'
alias glg='journalctl -fu galaxy-gunicorn@* | grep -v -e "/api/upload/hooks" -e "/history/current_history_json"'
alias glh='journalctl -f -u galaxy-handler@*'
alias glw='journalctl -f -u galaxy-workflow-scheduler@*'
alias glc='journalctl -fu galaxy-celery@*'
alias cu='journalctl -u galaxy-gunicorn@*.service --since "10 minutes ago" | grep "/history/current_history_json" | awk "{print \$11}" | sort -u | wc -l'
alias chg2wd='change_to_wd'
alias chg2jwd='change_to_jwd'

- name: Insert some export vars
lineinfile:
path: "{{ galaxy_user.home }}/.bashrc"
line: "export GALAXY_CONFIG_FILE={{ galaxy_config_file }}"
- name: Insert some export vars
lineinfile:
path: "{{ user_home }}/.bashrc"
# line: "export GALAXY_CONFIG_FILE={{ galaxy_config_file }}"
line: "{{ item }}"
loop:
# ENV's for gxadmin
- "export GALAXY_CONFIG_DIR={{ galaxy_config_dir }}"
- "export GALAXY_CONFIG_FILE={{ galaxy_config_file }}"
- "export GALAXY_LOG_DIR={{ galaxy_log_dir }}"
- "export GALAXY_MUTABLE_CONFIG_DIR={{ galaxy_mutable_config_dir }}"
- "export GALAXY_ROOT={{ galaxy_server_dir }}"
- "export VIRTUAL_ENV={{ galaxy_venv_dir }}"

- name: Check for bash_history
stat:
path: "{{ galaxy_user.home }}/.bash_history"
register: bashhistory_stat_out
- name: Check for bash_history
stat:
path: "{{ user_home }}/.bash_history"
register: bashhistory_stat_out

- name: Create bash_history
file:
path: "{{ galaxy_user.home }}/.bash_history"
state: touch
mode: 0640
owner: "{{ galaxy_user.name }}"
group: "{{ galaxy_group.name | default(galaxy_group) }}"
when: not bashhistory_stat_out.stat.exists
- name: Create bash_history
file:
path: "{{ user_home }}/.bash_history"
state: touch
mode: 0640
owner: "{{ user_name }}"
group: "{{ group_name }}"
when: not bashhistory_stat_out.stat.exists
when: (user_name in getent_passwd.keys())
61 changes: 61 additions & 0 deletions roles/usegalaxy-eu.bashrc/templates/get_jwd.sh.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/bin/bash
# Description: Get the job working directory using the job id and the object store configuration XML file
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have a python script for that, isn't it? How does this script relate to it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a Python script for cleaning up JWD's. I can also tweak that script and this functionality to it, and we can call a sub-command get_jwd via the command line that would return the JWD path and another sub-command cleanup_jwd that would clean up the JWDs if you would like.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would make sense to have this logic only in one place and not in two different scripts. Especially if you are planning to upstream your python script, we could reuse it easily.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have totally updated the script. Here is the gist. Please have a look and share your review.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sanjaysrikakulam this looks largely ok, just a few small things. I guess its ready for trying to upstream it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall I remove this bash script and replace it with the python script (in the above PR) until the upstream merge it and release it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have updated this PR. Please review everything and let me know if any changes are required.


# Get the object_store_id of the job from the 'galaxy' postgres database table 'job'
get_object_store_id() {
job_id=$1
object_store_id=$(psql -t -c "SELECT object_store_id FROM job WHERE id=$job_id AND object_store_id IS NOT NULL;")

# Check if object_store_id is empty
if [ -z "$object_store_id" ]; then
echo "ERROR: object_store_id not found for job_id '$job_id' in the 'galaxy' database"
exit 1
fi

echo $object_store_id
}

# Parse the object_store_conf.xml file and get the base path of the backend
get_base_path() {
object_store_conf_file=$1
object_store_id=$2
base_path=$(sed -n "/<backend id=\"$object_store_id\"/,/<\/backend>/p" $object_store_conf_file | grep -e '<extra_dir type="job_work"' | sed 's/.*<extra_dir type="job_work" path="\([^"]*\)".*/\1/')

# Check if base_path is empty
if [ -z "$base_path" ]; then
echo "ERROR: job_work path not found for object_store_id '$object_store_id' in the object_store_conf.xml file '$object_store_conf_file'"
exit 1
fi

echo "$base_path"
}

# Deconstruct the job id and get the job working directory path
get_jwd_path() {
job_id=$1
object_store_conf_file=$2
object_store_id=$(get_object_store_id $job_id)
base_path=$(get_base_path $object_store_conf_file $object_store_id)
jwd_path="$base_path/0${job_id:0:2}/${job_id:2:3}/$job_id"

# Check if jwd_path exists
if [ ! -d "$jwd_path" ]; then
echo "Job working directory (of $job_id) does not exist"
exit 1
fi

echo $jwd_path
}

# Main
job_id=$1
object_store_conf_file="{{ object_store_config_file }}"

# Check if object_store_conf_file exists
if [ ! -f "$object_store_conf_file" ]; then
echo "ERROR: Galaxy object store config file ($object_store_conf_file) does not exist"
exit 1
fi

jwd_path=$(get_jwd_path $job_id $object_store_conf_file)
echo $jwd_path