Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run nb in parallel #24

Merged
merged 1 commit into from
Oct 30, 2023
Merged

Run nb in parallel #24

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions scripts/notebook_runner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#!/usr/bin/env -S bash -eu -o pipefail
# next line loads AA_TOKEN from .env file when running bash script locally. In CI this is not necessary since AA_TOKEN is environment variable.
[ -f .env ] && source .env
AA_TOKEN="$AA_TOKEN" jupyter nbconvert --to notebook --execute src/examples/*.ipynb
rm src/examples/*.nbconvert.ipynb
# Find all .ipynb files in the directory and pass them to xargs for parallel execution
rm --force --recursive src/examples/*.nbconvert.ipynb src/examples/.ipynb_checkpoints
find src/examples/ -name "*.ipynb" | xargs -I {} --max-procs 1 bash -c "AA_TOKEN=\"$AA_TOKEN\" poetry run jupyter nbconvert --to notebook --execute {}"
rm --force src/examples/*.nbconvert.ipynb