-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add conformance gate prow job #19173
Conversation
Welcome @zachmandeville! |
Hi @zachmandeville. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/hold |
/assign @justaugustus |
/ok-to-test |
@zz Go ahead and specify the command as your startup script. |
spec: | ||
containers: | ||
- name: apisnoop-gate | ||
image: zachboyofdestiny/snoopdb:gate |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we avoid using a personal image here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@feiskyer , This job now uses an image from gcr.io/k8s-staging-apisnoop
.
Do you have a spyglass or gcsweb uri for those? They expired |
- name: postgres | ||
containerPort: 5432 | ||
env: | ||
- name: K8S_JOB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW there are a number of standard environment variables present in kubernetes CI
https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md#job-environment-variables
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, @BenTheElder! I updated the underlying image to use a standard env var, and removed my redundant one. This is part of commit a0c307c
2 questions:
|
@spiffxp , I do! The logs are coming from jobs running at https://prow.cncf.io (note: this job is now failing as 1.20.0 has a new, untested stable endpoint with |
To answer your questions, @aojea:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments/questions
/label tide/merge-method-squash |
137fbba
to
69e5b7d
Compare
/cc @johnbelamaric |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM in general, please update to use the test failures group alias I just created.
Co-authored-by: hh <[email protected]> change alert email to newly-made mailing list
d9742d7
to
96d7583
Compare
/test pull-kubernetes-node-crio-e2e |
@harche: The specified target(s) for
Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
sorry my bad. Wrong repo. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/approve
/lgtm
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dims, johnbelamaric, spiffxp, zachmandeville The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
@zachmandeville: Updated the
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Co-authored-by: hh <[email protected]> change alert email to newly-made mailing list
Summary
This PR introduces a job to check whether any stable endpoints promoted in the release are without conformance tests.
Background
It is a requirement that GA/stable API's have conformance tests. This requirement is hard to enforce and multiple releases have had stable endpoints promoted without tests. This creates technical debt for conformance,with test writers working on new tests to target API's from older releases. You can see this in this graph from APISnoop:
where the orange bars in each release mark newly promoted endpoints without tests, and the accumulation of technical debt.
This job
This prow job uses the same APISnoop database to pull in the audit logs from the most recent runs of these two e2e jobs:
It compares the endpoints hit by conformance tests in the audit logs to the endpoints defined in the open API spec, and lists any new GA endpoints that are not hit by a conformance test. If the list is empty, the job succeeds. If not, the job fails. One can then look at the bottom of the logs to find a list of untested endpoints.
You can see an example of this job on prow.cncf.io.
This job is intended to ultimately work as a release blocking job, and so its configured to fit the release blocking criteria.