add conformance gate prow job

[zachmandeville]

Summary

This PR introduces a job to check whether any stable endpoints promoted in the release are without conformance tests.

Background

It is a requirement that GA/stable API's have conformance tests. This requirement is hard to enforce and multiple releases have had stable endpoints promoted without tests. This creates technical debt for conformance,with test writers working on new tests to target API's from older releases. You can see this in this graph from APISnoop:

where the orange bars in each release mark newly promoted endpoints without tests, and the accumulation of technical debt.

This job

This prow job uses the same APISnoop database to pull in the audit logs from the most recent runs of these two e2e jobs:

• ci-kubernetes-e2e-gci-gce
• ci-kubernetes-gce-conformance-latest
  It compares the endpoints hit by conformance tests in the audit logs to the endpoints defined in the open API spec, and lists any new GA endpoints that are not hit by a conformance test. If the list is empty, the job succeeds. If not, the job fails. One can then look at the bottom of the logs to find a list of untested endpoints.
  You can see an example of this job on prow.cncf.io.

This job is intended to ultimately work as a release blocking job, and so its configured to fit the release blocking criteria.

[k8s-ci-robot]

Welcome @zachmandeville!

It looks like this is your first PR to kubernetes/test-infra 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/test-infra has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @zachmandeville. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zachmandeville]

/hold
/area conformance

[zachmandeville]

/assign @justaugustus

[dims]

/ok-to-test

[hh]

@zz Go ahead and specify the command as your startup script.

[feiskyer]

could we avoid using a personal image here?

[zachmandeville]

@feiskyer , yep! We just put in a change to APISnoop so that the new database it uses can be a part of the existing test-infra image builds for APISnoop. I opened a PR for this change here: #19234

When merged, I can update the container here to use this image.

[zachmandeville]

@feiskyer , This job now uses an image from gcr.io/k8s-staging-apisnoop.

[spiffxp]

You can see an example of this job on prow.cncf.io:

• the logs from a passing job( as 1.20.0 currently has no new untested endpoints)
• the logs from a purposefully failing job(it looks at untested endpoints from prior releases)

Do you have a spyglass or gcsweb uri for those? They expired

[BenTheElder]

FWIW there are a number of standard environment variables present in kubernetes CI
https://github.com/kubernetes/test-infra/blob/master/prow/jobs.md#job-environment-variables

[zachmandeville]

Good point, @BenTheElder! I updated the underlying image to use a standard env var, and removed my redundant one. This is part of commit a0c307c

[aojea]

2 questions:

1. can we have another job to gate per percentage? i.e. fail if GA less than 75%, beta less than 50%
2. can we use a different terminology to differentiate between API endpoints and the endpoints API?

[zachmandeville]

You can see an example of this job on prow.cncf.io:

• the logs from a passing job( as 1.20.0 currently has no new untested endpoints)
• the logs from a purposefully failing job(it looks at untested endpoints from prior releases)

Do you have a spyglass or gcsweb uri for those? They expired

@spiffxp , I do! The logs are coming from jobs running at https://prow.cncf.io
Here's a link to the logs for the most recent job

(note: this job is now failing as 1.20.0 has a new, untested stable endpoint with getInternalApiserverAPIGroup)

[zachmandeville]

2 questions:

1. can we have another job to gate per percentage? i.e. fail if GA less than 75%, beta less than 50%

2. can we use a different terminology to differentiate between API endpoints and the endpoints API?

To answer your questions, @aojea:

1. This would be straightforward to do with the foundations set in this job.
2. It would be straightforward to update the messages given in the logs to use different terminology, if this is what you mean?

[spiffxp]

Some comments/questions

[MushuEE]

/label tide/merge-method-squash

[spiffxp]

/cc @johnbelamaric
LGTM, but I'd like to get the /lgtm from @johnbelamaric as sig-arch chair

[johnbelamaric]

LGTM in general, please update to use the test failures group alias I just created.

[harche]

/test pull-kubernetes-node-crio-e2e

[k8s-ci-robot]

@harche: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

• /test pull-test-infra-bazel
• /test pull-test-infra-gubernator
• /test pull-test-infra-verify-file-perms
• /test pull-test-infra-yamllint
• /test pull-test-infra-prow-checkconfig

Use /test all to run all jobs.

In response to this:

/test pull-kubernetes-node-crio-e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[harche]

sorry my bad. Wrong repo.

[spiffxp]

/approve
/lgtm

[johnbelamaric]

/lgtm
/approve
/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, johnbelamaric, spiffxp, zachmandeville

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/OWNERS [dims,spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

@zachmandeville: Updated the job-config configmap in namespace default at cluster test-infra-trusted using the following files:

• key conformance-gate.yaml using file config/jobs/kubernetes/sig-arch/conformance-gate.yaml

In response to this:

Summary

This PR introduces a job to check whether any stable endpoints promoted in the release are without conformance tests.

Background

It is a requirement that GA/stable API's have conformance tests. This requirement is hard to enforce and multiple releases have had stable endpoints promoted without tests. This creates technical debt for conformance,with test writers working on new tests to target API's from older releases. You can see this in this graph from APISnoop:

where the orange bars in each release mark newly promoted endpoints without tests, and the accumulation of technical debt.

This job

This prow job uses the same APISnoop database to pull in the audit logs from the most recent runs of these two e2e jobs:

• ci-kubernetes-e2e-gci-gce
• ci-kubernetes-gce-conformance-latest
  It compares the endpoints hit by conformance tests in the audit logs to the endpoints defined in the open API spec, and lists any new GA endpoints that are not hit by a conformance test. If the list is empty, the job succeeds. If not, the job fails. One can then look at the bottom of the logs to find a list of untested endpoints.
  You can see an example of this job on prow.cncf.io.

This job is intended to ultimately work as a release blocking job, and so its configured to fit the release blocking criteria.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.