Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sdap 1223 make toil choice more dynamic #14

Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open

Sdap 1223 make toil choice more dynamic #14

wants to merge 35 commits into from

Conversation

scrowley-Datirium
Copy link

@scrowley-Datirium scrowley-Datirium commented May 20, 2024 •
edited
Loading

Description

1179: initial setup
1201: update configs
1209: fix cwl 'format' issue with toil
1213: remove 'listing' from outputs to reduce size
1222: add progress reporting script on cron job paired with run_toil
1223:

  • add batch_system as a config for cluster-sat ('single_machine' vs 'lsf' for cluster)
  • alter how progress reporting script works so processes aren't left hanging, and success/failures are properly reported

Closes issue(s)

https://datirium.atlassian.net/browse/SDAP-1179
https://datirium.atlassian.net/browse/SDAP-1201
https://datirium.atlassian.net/browse/SDAP-1209
https://datirium.atlassian.net/browse/SDAP-1213
https://datirium.atlassian.net/browse/SDAP-1222

https://datirium.atlassian.net/browse/SDAP-1223

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist

  • This code meets the definition of done
  • I have self-reviewed my own code
  • I have commented my code
  • I have made corresponding changes to the documentation

@scrowley-Datirium
setup everything for building cluster sat. issue with using conda thr…
7d074f4 
…ough script
@scrowley-Datirium
make tar gz with updated scripts for cluster satellite
1ac126b
@scrowley-Datirium
update readme
fa2d8a1
@scrowley-Datirium
add script for building normal satellite where njs client is build fr…
7a6deba 
…om local repo (need to alter bitbucket usage to github in normal build script)
@scrowley-Datirium
update build scripts
987786f
@scrowley-Datirium
formalize env setting and configuration for cluster-api within satell…
1708692 
…ite. (still need to make cluster-api repo public and make releases for download and setup)
@scrowley-Datirium
formalize more of the cluster sat configs
6ddce3b
@scrowley-Datirium
update cluster build scripts to get cluster-api from public git repo …
ad7a6b3 
…releases, instead of building from local repo
@scrowley-Datirium
start adding docs on replacing bitbucket access with github access. P…
b27716b 
…AT's may not be the best way (might want to copy ssh keys and establish github.com and known-host)
@scrowley-Datirium
modify cluster build script
7cb498a
@scrowley-Datirium
fix typo
79ed6c7
@scrowley-Datirium
fix typo in build script. alter configs to give njs port to cluster-api
7cd8b36
@scrowley-Datirium
fix typo in configure js. include dynamic njs-port for run_toil script
a231f10
@scrowley-Datirium
include path to config file being relative to satellite install locat…
1a92984 
…ion for cluster sat. make local build script use cwl-airflow fork instead of barski-lab
@scrowley-Datirium
update configs to include new run_toil param in sat settings. require…
d1237b9 
…s rebuilding cluster-api and cluster-sat
@scrowley-Datirium
SDAP-1209 fix format issue with toil
4fdb83f
@scrowley-Datirium
update default configs, configure js, and run toil for cluster satell…
069ce9f 
…ite (satellite now includes system PATH env in path for pm2 services)
@scrowley-Datirium
SDAP-1213 update build script for next version of cluster satellite
6a1b353
@scrowley-Datirium
update script to use barkilab instead of custom cwl-airflow fork
ef1c54a
@scrowley-Datirium
update script to use barkilab instead of custom cwl-airflow fork
cc87916
@scrowley-Datirium
setup progress script for toil
50eac6f
@scrowley-Datirium
update toil progress script so it's ready to be run in real sat
03de808
@scrowley-Datirium
add batch system as a config for cluster-sat. run toil script still n…
73263b8 
…eeds to use that config for dynamic toil runs
@scrowley-Datirium
update configs and run_toil script to handle extra configs for toil
8fa9a2f
@scrowley-Datirium
alter run_toil to dynamicaly call toil based on batch_system
61fea44
@scrowley-Datirium
update version tags for new builds
544d72d
@scrowley-Datirium
update versions for new cluster-api. remove postgres from cluster-sat…
2cdc392 
…ellite
@scrowley-Datirium
add system root to params passed to cluster-sat and toil script so jo…
ee28f1c 
…b file can have locations replaced properly
@scrowley-Datirium
fix bugs
445a301
@scrowley-Datirium
fix progress reporting
42b90cd
@scrowley-Datirium scrowley-Datirium changed the base branch from SDAP-1222-cluster-progress to main May 20, 2024 21:01
@scrowley-Datirium scrowley-Datirium mentioned this pull request May 21, 2024
Closed
@scrowley-Datirium scrowley-Datirium marked this pull request as ready for review May 21, 2024 18:42
This was referenced May 21, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@scrowley-Datirium