Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: cache test images for integration, VM and module tests #7599

Merged
merged 6 commits into from
Sep 26, 2024
Merged

ci: cache test images for integration, VM and module tests #7599

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c75cead
ci(test): cache test images for integration tests
DmitriyLewen Sep 26, 2024
49461b0
ci(test): use trivy-test-images last update date for cache key
DmitriyLewen Sep 26, 2024
208edcd
ci(test): cache test VM images
DmitriyLewen Sep 26, 2024
620ca42
ci(test): cache test images for module tests
DmitriyLewen Sep 26, 2024
c015645
refactor: use digest of image list for cache key
DmitriyLewen Sep 26, 2024
0609e5d
refactor: remove download steps
DmitriyLewen Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,26 @@ jobs:
with:
aqua_version: v1.25.0

- name: Generate image list digest
id: image-digest
run: |
IMAGE_LIST=$(skopeo list-tags docker://ghcr.io/aquasecurity/trivy-test-images)
DIGEST=$(echo "$IMAGE_LIST" | sha256sum | cut -d' ' -f1)
echo "digest=$DIGEST" >> $GITHUB_OUTPUT

- name: Restore test images from cache
uses: actions/cache@v4
id: restore-test-images
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: We no longer need id.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to merge this PR soon. We can delete it later.

with:
path: integration/testdata/fixtures/images
key: cache-test-images-${{ steps.image-digest.outputs.digest }}
restore-keys:
cache-test-images-

- name: Download test images
if: steps.restore-test-images.outputs.cache-hit != 'true'
knqyf263 marked this conversation as resolved.
Show resolved Hide resolved
run: mage test:fixtureContainerImages

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this step anymore. It is transparently called.

trivy/magefiles/magefile.go

Line 277 in bbc8e1d

mg.Deps(t.FixtureContainerImages)

And it's efficient as it skips images if it exists locally.

trivy/magefiles/fixture.go

Lines 30 to 32 in bbc8e1d

if exists(filePath) {
continue
}

Now, we call this check twice, which is just time-consuming (and API-consuming).

Copy link
Contributor Author

@DmitriyLewen DmitriyLewen Sep 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, since the cache will still be saved at the end of the job!

Removed download steps in 0609e5d

- name: Run integration tests
run: mage test:integration

Expand Down Expand Up @@ -122,6 +142,26 @@ jobs:
with:
aqua_version: v1.25.0

- name: Generate image list digest
id: image-digest
run: |
IMAGE_LIST=$(skopeo list-tags docker://ghcr.io/aquasecurity/trivy-test-images)
DIGEST=$(echo "$IMAGE_LIST" | sha256sum | cut -d' ' -f1)
echo "digest=$DIGEST" >> $GITHUB_OUTPUT

- name: Restore test images from cache
uses: actions/cache@v4
id: restore-test-images
with:
path: integration/testdata/fixtures/images
key: cache-test-images-${{ steps.image-digest.outputs.digest }}
restore-keys:
cache-test-images-

- name: Download test images
if: steps.restore-test-images.outputs.cache-hit != 'true'
run: mage test:fixtureContainerImages

- name: Run module integration tests
shell: bash
run: |
Expand All @@ -142,6 +182,27 @@ jobs:
uses: aquaproj/[email protected]
with:
aqua_version: v1.25.0

- name: Generate image list digest
id: image-digest
run: |
IMAGE_LIST=$(skopeo list-tags docker://ghcr.io/aquasecurity/trivy-test-vm-images)
DIGEST=$(echo "$IMAGE_LIST" | sha256sum | cut -d' ' -f1)
echo "digest=$DIGEST" >> $GITHUB_OUTPUT

- name: Restore test VM images from cache
uses: actions/cache@v4
id: restore-test-vm-images
with:
path: integration/testdata/fixtures/vm-images
key: cache-test-vm-images-${{ steps.image-digest.outputs.digest }}
restore-keys:
cache-test-vm-images-

- name: Download test VM images
if: steps.restore-test-vm-images.outputs.cache-hit != 'true'
run: mage test:fixtureVMImages

- name: Run vm integration tests
run: |
mage test:vm
Expand Down