Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add e2e tests #32

Closed
wants to merge 2 commits into from
Closed

Conversation

BobyMCbobs
Copy link
Member

Adds some e2e tests to target registry.k8s.io:

Related #24

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 29, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: BobyMCbobs
To complete the pull request process, please assign thockin after the PR has been reviewed.
You can assign the PR to them by writing /assign @thockin in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Mar 29, 2022
Copy link
Member

@BenTheElder BenTheElder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what I'm actually looking for in #24 is mentioned by this part:

We should then make sure we have k8s e2e jobs that use registry-sandbox, and have them in a testgrid dashboard for this project.

We should have actual kubernetes e2e tests (like kubetest + gce) running in a testgrid dashboard for this project with alerting (conformance tests maybe? or just the parallel subset?) but using registry-sandbox.k8s.io as the kubernetes image registry with the latest code from this repo deployed to registry-sandbox.k8s.io (#24 (comment) => kubernetes/k8s.io#3411), so we know even the cloud run deployment etc is configured correctly, and that it works with a real client we can expect.

I think this will drive far higher confidence that we're ready to promote a change from the sandbox to production.

In #23 I think we can write something reasonable with a shell script + crane to spot check running a local registry binary and pull a known small image (pause?) through it, and add this as a make integration rule, which will give us a quicker way to test changes while developing / in presubmit, though with lower confidence. We can track crane in the hack/tools module and build and run an archeio binary locally with the make targets.

@@ -2,6 +2,25 @@ module sigs.k8s.io/oci-proxy

go 1.17

require k8s.io/klog/v2 v2.40.1
require (
github.com/google/go-containerregistry v0.8.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's move the test tool to it's own module, so we can clearly keep imports minimal in the main binary and not require downloading + installing test-only deps during building.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(or, perhaps rework entirely 😅 #32 (review))

@BenTheElder
Copy link
Member

we sort of have those e2e jobs today, but instead they're not in a dashboard we're tracking and they're using the production domain ahead of us rolling that out more widely, we're missing "auto deploy the latest code to registry-sandbox" and "run some realistic CI jobs against registry-sandbox".

I think for running against the deployed registry, we should skip straight to an actual cluster pulling from it, and only use containerregistry/crane as the client for local integration tests where it's not as reasonable to use a cluster / we're not covering the deployment configs anyhow.

@BenTheElder
Copy link
Member

thanks for the PR, I've added an integration test in #47, we should setup the auto-deployed sandbox with e2e test clusters running against it in a dashboard in #24.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
Development

Successfully merging this pull request may close these issues.

3 participants