Cache LFS objects in the tests workflow

[atharva-2001]

📝 Description

Type: 🪲 bugfix | 🚀 feature | ☣️ breaking change | 🚦 testing | 📝 documentation | 🎢 infrastructure

Write a complete description of your changes, including the necessary context or any piece of information required to understand your work.

Also, link issues affected by this pull request by using the keywords: close, closes, closed, fix, fixes, fixed, resolve, resolves or resolved.

📌 Resources

Examples, notebooks, and links to useful references.

🚦 Testing

How did you test these changes?

• Testing pipeline
• Other method (describe)
• My changes can't be tested (explain why)

☑️ Checklist

• I requested two reviewers for this pull request
• I updated the documentation according to my changes
• I built the documentation by applying the build_docs label

Note: If you are not allowed to perform any of these actions, ping (@) a contributor.

[tardis-bot]

*beep* *bop*

Hi, human.

I'm the @tardis-bot and couldn't find your records in my database. I think we don't know each other, or you changed your credentials recently.

Please add your name and email to .mailmap in your current branch and push the changes to this pull request.

In case you need to map an existing alias, follow this example.

[codecov]

Codecov Report

Merging #2194 (38fe960) into master (5e107ea) will not change coverage.
The diff coverage is n/a.

❗ Current head 38fe960 differs from pull request most recent head 8e68779. Consider uploading reports for the commit 8e68779 to get more accurate results

@@           Coverage Diff           @@
##           master    #2194   +/-   ##
=======================================
  Coverage   71.81%   71.81%           
=======================================
  Files         133      133           
  Lines       12355    12355           
=======================================
  Hits         8873     8873           
  Misses       3482     3482           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[sonachitchyan]

@atharva-2001 be sure to add your info into the .mailmap

[tardis-bot]

*beep* *bop*

Hi, human.

I'm the @tardis-bot and couldn't find your records in my database. I think we don't know each other, or you changed your credentials recently.

Please add your name and email to .mailmap in your current branch and push the changes to this pull request.

In case you need to map an existing alias, follow this example.

[atharva-2001]

The tests here are not failing, but the tests here are. I wonder why that is happening. However, the files are being cached!

[epassaro]

I don't think this is going to work as expected.

You need to use a conditional step, like we do in this example:

      - uses: actions/cache@v2
        with:
          path: ${{ matrix.prefix }}
          key: conda-${{ matrix.label }}-${{ hashFiles('conda-${{ matrix.label }}.lock') }}-${{ env.CACHE_NUMBER }}
        id: cache-conda

      - name: Update environment
        run: mamba update -n tardis --file conda-${{ matrix.label }}.lock
        if: steps.cache-conda.outputs.cache-hit != 'true'

In the example, the second step only runs if the cache exists. In your code, all the steps are running on every commit.

I think this could be achieved by doing:

1. Run the cache step.
2. Clone the repo without the LFS objects (there's an option for that)
3. Get the LFS assets (I'm assuming this step is going to work independently of pulling the LFS objects, but I don't know)
4. Make a conditional step that pulls the LFS objects if the cache does not exist.

[atharva-2001]

I think this could be achieved by doing:

1. Run the cache step.
2. Clone the repo without the LFS objects (there's an option for that)
3. Get the LFS assets (I'm assuming this step is going to work independently of pulling the LFS objects, but I don't know)
4. Make a conditional step that pulls the LFS objects if the cache does not exist.

But this is what I'm doing here right? Here is what I think the workflow does:

1. Clone the repo without LFS Objects
2. Generate a file(.lfs-assets-id) with all the hashes of the file names. This file is then used to generate the cache key. If any of these files change, then the key would be invalid.
3. If the cache key matches, the items would be restored and git lfs pull wouldn't do anything. If it fails, there would be nothing in the lfs folder and the items would be downloaded(that way we won't need a conditional statement.)

[tardis-bot]

*beep* *bop*

Hi, human.

I'm the @tardis-bot and couldn't find your records in my database. I think we don't know each other, or you changed your credentials recently.

Please add your name and email to .mailmap in your current branch and push the changes to this pull request.

In case you need to map an existing alias, follow this example.

[tardis-bot]

*beep* *bop*

Hi, human.

I'm the @tardis-bot and couldn't find your records in my database. I think we don't know each other, or you changed your credentials recently.

Please add your name and email to .mailmap in your current branch and push the changes to this pull request.

In case you need to map an existing alias, follow this example.

[epassaro]

I think you are right @atharva-2001, but I think we should keep the lfs: false flag and the conditional statement explicit if the result is the same, just to be more clear in case we need to revisit this in the future.

[tardis-bot]

*beep* *bop*

Hi, human.

I'm the @tardis-bot and couldn't find your records in my database. I think we don't know each other, or you changed your credentials recently.

Please add your name and email to .mailmap in your current branch and push the changes to this pull request.

In case you need to map an existing alias, follow this example.

[andrewfullard]

I think you are right @atharva-2001, but I think we should keep the lfs: false flag and the conditional statement explicit if the result is the same, just to be more clear in case we need to revisit this in the future.

Why lfs: false? Confusing to me.

[atharva-2001]

Why lfs: false? Confusing to me.

The LFS objects would be saved as cache and we wouldn't need to get them every time the workflow runs. We would only need to get LFS objects if the reference data repository is updated(because that would invalidate the cache key) and we are doing that using git lfs pull.

[chvogl]

Looks good to me!

[chvogl]

@atharva-2001 Any idea why the tests are failing?

[atharva-2001]

I'm afraid I'm yet to look at it. I'd soon fix this and let you know!

[chvogl]

Looks great!