Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache LFS Objects in the TARDIS-Carsus Compatibility Check Workflow #327

Merged
merged 63 commits into from
Mar 30, 2023
Merged

Cache LFS Objects in the TARDIS-Carsus Compatibility Check Workflow #327

Changes from 62 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
71f9239
Initial commit
atharva-2001 Jun 28, 2022
46591b5
Add step to download the CMFGEN database
atharva-2001 Jun 29, 2022
f91b7fb
Add URLs to the CMFGEN database
atharva-2001 Jun 29, 2022
d8232b0
Add tardis build job
atharva-2001 Jun 29, 2022
e9487e4
Add strategy for TARDIS build job
atharva-2001 Jun 30, 2022
3fd40c0
Fix lock file path
atharva-2001 Jun 30, 2022
0983b61
Add step to run notebook
atharva-2001 Jun 30, 2022
0e69c0f
Upload spectrum notebook
atharva-2001 Jun 30, 2022
b211146
Fix atom data path
atharva-2001 Jul 3, 2022
3e1d648
Change URL used to get the reference data file
atharva-2001 Jul 10, 2022
db03395
Download the file from the workflow
atharva-2001 Jul 10, 2022
3715561
Upload reference data as artifact
atharva-2001 Jul 14, 2022
38a8ce3
Try downloading all reference data
atharva-2001 Jul 18, 2022
6a419d7
Clone my fork of tardis refdata
atharva-2001 Jul 20, 2022
85208f1
Use lfs when cloning
atharva-2001 Jul 20, 2022
7ed5bca
Separate commands
atharva-2001 Jul 20, 2022
6a14e28
Add new env variables, generate reference data for comparision using …
atharva-2001 Jul 20, 2022
f0552c6
Clone my fork with arraydiff unpinned
atharva-2001 Jul 21, 2022
cda1c5e
Fix notebook path when creating artifact
atharva-2001 Jul 23, 2022
c100e69
Fix paths in environment variables
atharva-2001 Jul 24, 2022
b808e38
Install bokeh
atharva-2001 Jul 24, 2022
0eed8c0
Fix paths after moving branch
atharva-2001 Jul 26, 2022
79a26cf
Rename notebook
atharva-2001 Jul 28, 2022
890c619
Remove variables related to matrix strategy
atharva-2001 Jul 28, 2022
40b8496
Remove step to download reference data
atharva-2001 Jul 28, 2022
fbf5c56
Create copy of the quickstart notebook
atharva-2001 Jul 29, 2022
b8129a3
Delete markdown
atharva-2001 Jul 29, 2022
b820d3f
Add refdata_gen notebook instead of quickstart
atharva-2001 Jul 29, 2022
f0abdf9
Change branch when cloning carsus in the workflow
atharva-2001 Jul 29, 2022
6fb9add
Try removing chianti reader
atharva-2001 Aug 2, 2022
ca692c5
Put back elements in the atomic data
atharva-2001 Aug 2, 2022
338b266
Add chianti data back but without collisions
atharva-2001 Aug 2, 2022
d5c9b19
Change Chianti version
atharva-2001 Aug 3, 2022
56ee1ef
Continue workflow if step fails
atharva-2001 Aug 8, 2022
da36d02
Revert back Chianti version
atharva-2001 Aug 8, 2022
b16c783
Add step to fail job if spectra dont match
atharva-2001 Aug 11, 2022
91174d8
test
atharva-2001 Aug 11, 2022
e95c2e2
Exit if contents of tmp file are set to 0
atharva-2001 Aug 11, 2022
96f0c54
Exit code
atharva-2001 Aug 11, 2022
5917f4f
revert comments
atharva-2001 Aug 11, 2022
6cbe0c5
On workflow dispatch
atharva-2001 Aug 16, 2022
42c2bb4
Rename steps and use variables
atharva-2001 Sep 8, 2022
ecbf35c
Replace NBCONVERT_CMD variable with NBCONVERT_FLAGS
atharva-2001 Oct 19, 2022
b42d57f
Comment out Carsus build job temporarily, download reference data fro…
atharva-2001 Oct 20, 2022
b2d15a0
Disable workflow dispatch temporarily
atharva-2001 Oct 20, 2022
994066a
Fix paths for refdata compare notebook
atharva-2001 Oct 20, 2022
d0fef1f
Carsus build is necessary for TARDIS build
atharva-2001 Oct 20, 2022
8f920ad
Change file name in notebook
atharva-2001 Oct 20, 2022
f9b7183
Delete lines commented out which are not necessary
atharva-2001 Oct 20, 2022
0da5cbb
Do not clone my fork of the carsus repository
atharva-2001 Oct 25, 2022
57be609
Change title of the notebook and rename it
atharva-2001 Oct 26, 2022
9eb8aa8
Change file and step names
atharva-2001 Oct 26, 2022
db6cfd2
No need for inputs
atharva-2001 Oct 27, 2022
80d1417
Check file contents according to changes in tardis-sn/tardis-refdata#58
atharva-2001 Oct 30, 2022
dafafd3
Change path of refdata_compare_result file
atharva-2001 Nov 8, 2022
6fa893a
Change trigger back to workflow_dispatch
atharva-2001 Nov 9, 2022
c7aaaeb
Update URL to use TARDIS fork
atharva-2001 Nov 17, 2022
a5cd46a
Do not pin bokeh
atharva-2001 Jan 12, 2023
3a8c50c
Add name to comparison notebook artifact
atharva-2001 Jan 12, 2023
90e8ac7
Cache LFS objects
atharva-2001 Jan 27, 2023
5cb8396
Do git lfs checkout if the cache key is found
atharva-2001 Feb 23, 2023
07f4570
Merge branch 'master' into cache_LFS_bridge
atharva-2001 Mar 9, 2023
c22ea9f
Use cache no instead of v1 hardcoding
atharva-2001 Mar 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions .github/workflows/bridge.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,8 +91,32 @@ jobs:
- name: Download refdata_compare notebook
run: wget https://raw.githubusercontent.com/tardis-sn/tardis-refdata/master/notebooks/ref_data_compare_from_paths.ipynb

- name: Download reference data
run: bash .ci-helpers/download_reference_data.sh
- name: Clone tardis-sn/tardis-refdata
uses: actions/checkout@v2
with:
repository: tardis-sn/tardis-refdata
path: tardis-refdata

- name: Create LFS file list
run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id
working-directory: tardis-refdata

- name: Restore LFS cache
uses: actions/cache@v2
id: lfs-cache
with:
path: tardis-refdata/.git/lfs
key: ${{ runner.os }}-lfs-${{ hashFiles('tardis-refdata/.lfs-assets-id') }}-v1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why v1 is hardcoded?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case we want to invalidate the cache manually- actions/checkout#165

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we use a CACHE_NUMBER: 0 variable on top of the file for that (I think setting to 1 is better though) and then replace v1 with v${{ env.CACHE_NUMBER }}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @epassaro.


- name: Git LFS Pull
run: git lfs pull
working-directory: tardis-refdata
if: steps.lfs-cache.outputs.cache-hit != 'true'

- name: Git LFS Checkout
run: git lfs checkout
working-directory: tardis-refdata
if: steps.lfs-cache.outputs.cache-hit == 'true'

- name: Setup TARDIS environment
uses: conda-incubator/setup-miniconda@v2
Expand Down