Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update chain's controllers to use v1 Tekton APIs natively while converting to v1beta1 to keep formats backwards compatible #1016

Merged
merged 1 commit into from
Jan 17, 2024

Conversation

aaron-prindle
Copy link
Contributor

@aaron-prindle aaron-prindle commented Dec 14, 2023

fixes #665 , fixes #985, fixes #986

This PR updates chains to natively use the Tekton Pipeline v1 API ("github.com/tektoncd/pipeline/pkg/apis/pipeline/v1"), updated from the Tekton Pipeline v1beta1 API ("github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1"). Internally in the provenance and signing logic v1 Tekton object representation is used w/ v1beta1 objects being converted in/out of v1 as needed for processing fields and storing provenance (to keep it as similar as possible for users upgrading chains)

The backwards compatibility works as v1beta1 and v1 Tekton APIs can be converted between types. Due to v1beta1 changes over time, some fields have their own bespoke support (v1beta1 TaskRun .Spec.Resources, and .Status.TaskRunStatusField.TaskSpec.Resources).

For a V1 object using the V1 controller, the flow is straightforward like it was initially with V1 objects being used throughout the signing process

For a V1Beta1 object using the v1beta1 controller, the flow has 4 main steps:

  1. The received V1Beta1 Tekton Object ([PipelineRun|TaskRun]) is converted to a V1 Tekton Object adding some additional serialized fields to annotations that chains might need to use later when re-converting.
  2. When certain extraction/signing logic is hit that needs to check .Spec.Resources, .Spec.ResourceResults, or .Status.TaskRunStatusField.TaskSpec.Resources we convert the V1 object back to V1Beta1 (using Tekton Pipeline conversion lib + bespoke deserialization code) and extract the necessary information as was done prior. Then the V1 Provenance is uploaded.

The changes required were broadly the following:

  • s/v1beta1/v1 such that all internal logic uses v1 directly (except for edge cases of converting to v1beta1 to get a field from Pipeline Conversion webhook annotations AND converting v1beta1 for SLSA format compatibility in v1, v2, v2alpha1 & v2alpha2)
    • where v1beta1 fields were used to generate provenance, add logic to convert back to v1beta1 to extract fields when necessary
    • convert necessary test files from v1beta1 -> v1
  • update objects.go TektonObject interface to be more generic to support to v1beta1 and v1 objects (some interface types were versioned @ HEAD - Result & Provenance). Making the controller more natively support v1beta1 and v1 is non-trivial as in many places the TektonObject is cast back to a versioned type (eg: v1, or @ priort-to-this-pr v1beta1) for reading fields directly, etc From review it was preferred to keep these versioned

Open Questions:

  • Is it necessary to update storage/* to use v1beta1 as well as format/* (v1, v2alpha1, v2alpha2)?
    • Storage interface takes a TektonObject obj which is versioned -> StorePayload(ctx context.Context, obj objects.TektonObject, rawPayload []byte, signature string, opts config.StorageOpts) error
    • GVK stored as in grafeas storage opts
    • GCS seems to use versioned (eg: v1.[TaskRun|Pipeline] references) Tekton Pipeline libs
  • v1beta1 had it's API surface change over time with some fields that chains uses being deprecated ~2 years ago. Additionally chains looks at many v1beta1 fields through their .Status.* value vs their .Spec value as the .Status value is fully resolved from parameters, etc. In practice this means that using ConvertFrom to go v1 -> v1beta1 to pickup some of these old fields doesn't work as intended and this cannot simply be fixed by using the .Spec value (vs .Status) as they are different (one is fully resolved). Currently the submitted code introduces new annotations or .Spec to .Status mapping to keep as many pre-existing tests as possible as there were prior but likely either the Pipeline conversion functions should be updated upstream to allow for this backwards compat OR the custom annotation logic in the PR here should be dropped outright. This would likely mean removing some of these old and likely deprecated tests as well

@tekton-robot tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Dec 14, 2023
@JeromeJu
Copy link
Member

/assign @chitrangpatel
/assign

@tekton-robot
Copy link

@tekton-robot
Copy link

Copy link
Contributor

@chitrangpatel chitrangpatel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this massive effort 🙏 .
Left some comments.

Just to confirm my high level significant changes:
Controller:
only looks at v1 now. This means that it will receive v1 objects.

Signing, Storers etc.
Understands v1 (no longer v1beta1). Wherever there was a use of pipeline resources, we convert to v1beta1 and call the v1beta1 logic.

In formats:

  • v1: we convert to v1beta1 and analyze
  • v2alpha1: we convert to v1beta1
  • v2alpha2: convert to v1beta1
  • v2alpha3: no need for conversion. Will look at v1, which is what the controller receives.

@chitrangpatel
Copy link
Contributor

/assign @wlynch @lcarva PTAL 🙏 (specifically the storer and signing parts since I haven't used them a lot).

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 18, 2023
@chitrangpatel
Copy link
Contributor

/assign @wlynch @lcarva

PTAL when you have a chance since this is a large PR.

Thanks 🙏

@chitrangpatel
Copy link
Contributor

/approve

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/artifacts/signable.go 70.2% 70.4% 0.2
pkg/chains/formats/slsa/extract/extract.go 61.5% 62.3% 0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go Do not exist 61.5%
pkg/chains/formats/slsa/internal/material/material.go 92.2% 92.3% 0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go Do not exist 92.2%
pkg/chains/formats/slsa/v1/intotoite6.go 88.9% 30.8% -58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go 85.7% 50.0% -35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go 87.5% 28.0% -59.5
pkg/chains/objects/objects.go 69.7% 32.7% -37.0
pkg/reconciler/pipelinerun/controller.go 95.0% 86.4% -8.6
pkg/reconciler/pipelinerun/pipelinerun.go 80.5% 82.9% 2.4
pkg/reconciler/taskrun/controller.go 94.1% 88.9% -5.2

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/artifacts/signable.go 70.2% 70.4% 0.2
pkg/chains/formats/slsa/extract/extract.go 61.5% 62.3% 0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go Do not exist 61.5%
pkg/chains/formats/slsa/internal/material/material.go 92.2% 92.3% 0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go Do not exist 92.2%
pkg/chains/formats/slsa/v1/intotoite6.go 88.9% 30.8% -58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go 85.7% 50.0% -35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go 87.5% 28.0% -59.5
pkg/chains/objects/objects.go 69.7% 32.7% -37.0
pkg/reconciler/pipelinerun/controller.go 95.0% 86.4% -8.6
pkg/reconciler/pipelinerun/pipelinerun.go 80.5% 82.9% 2.4
pkg/reconciler/taskrun/controller.go 94.1% 88.9% -5.2

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/artifacts/signable.go 70.2% 70.4% 0.2
pkg/chains/formats/slsa/extract/extract.go 61.5% 62.3% 0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go Do not exist 61.5%
pkg/chains/formats/slsa/internal/material/material.go 92.2% 92.3% 0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go Do not exist 92.2%
pkg/chains/formats/slsa/v1/intotoite6.go 88.9% 30.8% -58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go 85.7% 50.0% -35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go 87.5% 28.0% -59.5
pkg/chains/objects/objects.go 69.7% 32.7% -37.0
pkg/chains/signing.go 73.7% 52.6% -21.2
pkg/reconciler/pipelinerun/controller.go 95.0% 86.4% -8.6
pkg/reconciler/pipelinerun/pipelinerun.go 80.5% 82.9% 2.4
pkg/reconciler/taskrun/controller.go 94.1% 88.9% -5.2

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/artifacts/signable.go 70.2% 70.4% 0.2
pkg/chains/formats/slsa/extract/extract.go 61.5% 62.3% 0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go Do not exist 61.5%
pkg/chains/formats/slsa/internal/material/material.go 92.2% 92.3% 0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go Do not exist 92.2%
pkg/chains/formats/slsa/v1/intotoite6.go 88.9% 30.8% -58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go 85.7% 50.0% -35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go 87.5% 28.0% -59.5
pkg/chains/objects/objects.go 69.7% 32.7% -37.0
pkg/reconciler/pipelinerun/controller.go 95.0% 86.4% -8.6
pkg/reconciler/pipelinerun/pipelinerun.go 80.5% 82.9% 2.4
pkg/reconciler/taskrun/controller.go 94.1% 88.9% -5.2

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/artifacts/signable.go 70.2% 70.4% 0.2
pkg/chains/formats/slsa/extract/extract.go 61.5% 62.3% 0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go Do not exist 61.5%
pkg/chains/formats/slsa/internal/material/material.go 92.2% 92.3% 0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go Do not exist 92.2%
pkg/chains/formats/slsa/v1/intotoite6.go 88.9% 30.8% -58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go 85.7% 50.0% -35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go 87.5% 28.0% -59.5
pkg/chains/objects/objects.go 69.7% 32.7% -37.0
pkg/chains/storage/gcs/gcs.go 70.1% 64.2% -5.9
pkg/reconciler/pipelinerun/controller.go 95.0% 86.4% -8.6
pkg/reconciler/pipelinerun/pipelinerun.go 80.5% 82.9% 2.4
pkg/reconciler/taskrun/controller.go 94.1% 88.9% -5.2

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/artifacts/signable.go 70.2% 70.4% 0.2
pkg/chains/formats/slsa/extract/extract.go 61.5% 62.3% 0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go Do not exist 61.5%
pkg/chains/formats/slsa/internal/material/material.go 92.2% 92.3% 0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go Do not exist 92.2%
pkg/chains/formats/slsa/v1/intotoite6.go 88.9% 30.8% -58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go 85.7% 50.0% -35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go 87.5% 28.0% -59.5
pkg/chains/objects/objects.go 69.7% 32.7% -37.0
pkg/chains/storage/gcs/gcs.go 70.1% 64.2% -5.9
pkg/reconciler/pipelinerun/controller.go 95.0% 86.4% -8.6
pkg/reconciler/pipelinerun/pipelinerun.go 80.5% 82.9% 2.4
pkg/reconciler/taskrun/controller.go 94.1% 88.9% -5.2

Copy link
Member

@wlynch wlynch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good! Some final questions, otherwise should be able to get this merged.

pkg/chains/storage/gcs/gcs.go Show resolved Hide resolved
pkg/reconciler/pipelinerun/pipelinerun.go Show resolved Hide resolved
… converting to v1beta1 to keep formats backwards compatible
@aaron-prindle
Copy link
Contributor Author

aaron-prindle commented Jan 11, 2024

Thanks for the reviews @wlynch and @lcarva! @wlynch I believe in the final review the only remaining work item was to update the godoc for gcs.go - StorePayload stating that the method was updated to use v1 (from v1beta1) and messages will reflect that. I have updated the godoc as such now:

// StorePayload implements the storage.Backend interface.  As of chains v0.20.0+,
// this method has been updated to use Tekton v1 objects (previously v1beta1) and
// it's error messages have been updated to reflect this.

With this change I believe this should be mergable IIUC. Thanks!

@tekton-robot
Copy link

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/artifacts/signable.go 70.2% 70.4% 0.2
pkg/chains/formats/slsa/extract/extract.go 61.5% 62.3% 0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go Do not exist 61.5%
pkg/chains/formats/slsa/internal/material/material.go 92.2% 92.3% 0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go Do not exist 92.2%
pkg/chains/formats/slsa/v1/intotoite6.go 88.9% 30.8% -58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go 85.7% 50.0% -35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go 87.5% 28.0% -59.5
pkg/chains/objects/objects.go 69.7% 32.7% -37.0
pkg/chains/storage/gcs/gcs.go 70.1% 64.2% -5.9
pkg/reconciler/pipelinerun/controller.go 95.0% 86.4% -8.6
pkg/reconciler/pipelinerun/pipelinerun.go 80.5% 82.9% 2.4
pkg/reconciler/taskrun/controller.go 94.1% 88.9% -5.2

Copy link
Member

@wlynch wlynch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jan 16, 2024
@chitrangpatel
Copy link
Contributor

/kind misc

@tekton-robot tekton-robot added the kind/misc Categorizes issue or PR as a miscellaneuous one. label Jan 17, 2024
@tekton-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chitrangpatel, JeromeJu, wlynch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [chitrangpatel,wlynch]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot merged commit 592410c into tektoncd:main Jan 17, 2024
17 checks passed
khrm pushed a commit to khrm/chains that referenced this pull request Jan 22, 2024
… converting to v1beta1 to keep formats backwards compatible (tektoncd#1016)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/misc Categorizes issue or PR as a miscellaneuous one. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
6 participants