Skip to content

Commit

Permalink
Run nb in parallel
Browse files Browse the repository at this point in the history
  • Loading branch information
volkerstampa committed Oct 30, 2023
1 parent 65aff82 commit 5b65bb1
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions scripts/notebook_runner.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/usr/bin/env -S bash -eu -o pipefail
# next line loads AA_TOKEN from .env file when running bash script locally. In CI this is not necessary since AA_TOKEN is environment variable.
[ -f .env ] && source .env
AA_TOKEN="$AA_TOKEN" jupyter nbconvert --to notebook --execute src/examples/*.ipynb
rm src/examples/*.nbconvert.ipynb
# Find all .ipynb files in the directory and pass them to xargs for parallel execution
rm --force --recursive src/examples/*.nbconvert.ipynb src/examples/.ipynb_checkpoints
find src/examples/ -name "*.ipynb" | xargs -I {} --max-procs 1 bash -c "AA_TOKEN=\"$AA_TOKEN\" poetry run jupyter nbconvert --to notebook --execute {}"
rm --force src/examples/*.nbconvert.ipynb

0 comments on commit 5b65bb1

Please sign in to comment.