feat: update chain's controllers to use v1 Tekton APIs natively while converting to v1beta1 to keep formats backwards compatible

[aaron-prindle]

fixes #665 , fixes #985, fixes #986

This PR updates chains to natively use the Tekton Pipeline v1 API ("github.com/tektoncd/pipeline/pkg/apis/pipeline/v1"), updated from the Tekton Pipeline v1beta1 API ("github.com/tektoncd/pipeline/pkg/apis/pipeline/v1beta1"). Internally in the provenance and signing logic v1 Tekton object representation is used w/ v1beta1 objects being converted in/out of v1 as needed for processing fields and storing provenance (to keep it as similar as possible for users upgrading chains)

The backwards compatibility works as v1beta1 and v1 Tekton APIs can be converted between types. Due to v1beta1 changes over time, some fields have their own bespoke support (v1beta1 TaskRun .Spec.Resources, and .Status.TaskRunStatusField.TaskSpec.Resources).

For a V1 object using the V1 controller, the flow is straightforward like it was initially with V1 objects being used throughout the signing process

For a V1Beta1 object using the v1beta1 controller, the flow has 4 main steps:

1. The received V1Beta1 Tekton Object ([PipelineRun|TaskRun]) is converted to a V1 Tekton Object adding some additional serialized fields to annotations that chains might need to use later when re-converting.
2. When certain extraction/signing logic is hit that needs to check .Spec.Resources, .Spec.ResourceResults, or .Status.TaskRunStatusField.TaskSpec.Resources we convert the V1 object back to V1Beta1 (using Tekton Pipeline conversion lib + bespoke deserialization code) and extract the necessary information as was done prior. Then the V1 Provenance is uploaded.

The changes required were broadly the following:

• s/v1beta1/v1 such that all internal logic uses v1 directly (except for edge cases of converting to v1beta1 to get a field from Pipeline Conversion webhook annotations AND converting v1beta1 for SLSA format compatibility in v1, v2, v2alpha1 & v2alpha2)
  • where v1beta1 fields were used to generate provenance, add logic to convert back to v1beta1 to extract fields when necessary
  • convert necessary test files from v1beta1 -> v1
• update objects.go TektonObject interface to be more generic to support to v1beta1 and v1 objects (some interface types were versioned @ HEAD - Result & Provenance). Making the controller more natively support v1beta1 and v1 is non-trivial as in many places the TektonObject is cast back to a versioned type (eg: v1, or @ priort-to-this-pr v1beta1) for reading fields directly, etc From review it was preferred to keep these versioned

Open Questions:

• Is it necessary to update storage/* to use v1beta1 as well as format/* (v1, v2alpha1, v2alpha2)?
  • Storage interface takes a TektonObject obj which is versioned -> StorePayload(ctx context.Context, obj objects.TektonObject, rawPayload []byte, signature string, opts config.StorageOpts) error
  • GVK stored as in grafeas storage opts
  • GCS seems to use versioned (eg: v1.[TaskRun|Pipeline] references) Tekton Pipeline libs
• v1beta1 had it's API surface change over time with some fields that chains uses being deprecated ~2 years ago. Additionally chains looks at many v1beta1 fields through their .Status.* value vs their .Spec value as the .Status value is fully resolved from parameters, etc. In practice this means that using ConvertFrom to go v1 -> v1beta1 to pickup some of these old fields doesn't work as intended and this cannot simply be fixed by using the .Spec value (vs .Status) as they are different (one is fully resolved). Currently the submitted code introduces new annotations or .Spec to .Status mapping to keep as many pre-existing tests as possible as there were prior but likely either the Pipeline conversion functions should be updated upstream to allow for this backwards compat OR the custom annotation logic in the PR here should be dropped outright. This would likely mean removing some of these old and likely deprecated tests as well

[JeromeJu]

/assign @chitrangpatel
/assign

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	71.6%	1.4
pkg/artifacts/structured.go	96.3%	96.4%	0.1
pkg/chains/formats/slsa/extract/extract.go	61.5%	63.2%	1.6
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/artifact/v1beta1/append.go	Do not exist	100.0%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.6%	0.4
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/formats/slsa/v2alpha3/internal/external_parameters/external_parameters.go	Do not exist	87.5%
pkg/chains/formats/slsa/v2alpha3/internal/internal_parameters/internal_parameters.go	Do not exist	100.0%
pkg/chains/formats/slsa/v2alpha3/internal/pipelinerun/pipelinerun.go	Do not exist	83.8%
pkg/chains/formats/slsa/v2alpha3/internal/resolved_dependencies/resolved_dependencies.go	Do not exist	87.5%
pkg/chains/formats/slsa/v2alpha3/internal/taskrun/taskrun.go	Do not exist	86.5%
pkg/chains/formats/slsa/v2alpha3/slsav2.go	Do not exist	87.5%
pkg/chains/objects/objects.go	69.7%	31.5%	-38.2
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	71.6%	1.4
pkg/artifacts/structured.go	96.3%	96.4%	0.1
pkg/chains/formats/slsa/extract/extract.go	61.5%	63.2%	1.6
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/artifact/v1beta1/append.go	Do not exist	100.0%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.6%	0.4
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/formats/slsa/v2alpha3/internal/external_parameters/external_parameters.go	Do not exist	87.5%
pkg/chains/formats/slsa/v2alpha3/internal/internal_parameters/internal_parameters.go	Do not exist	100.0%
pkg/chains/formats/slsa/v2alpha3/internal/pipelinerun/pipelinerun.go	Do not exist	83.8%
pkg/chains/formats/slsa/v2alpha3/internal/resolved_dependencies/resolved_dependencies.go	Do not exist	87.5%
pkg/chains/formats/slsa/v2alpha3/internal/taskrun/taskrun.go	Do not exist	86.5%
pkg/chains/formats/slsa/v2alpha3/slsav2.go	Do not exist	87.5%
pkg/chains/objects/objects.go	69.7%	31.5%	-38.2
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4

[chitrangpatel]

Thanks for this massive effort 🙏 .
Left some comments.

Just to confirm my high level significant changes:
Controller:
only looks at v1 now. This means that it will receive v1 objects.

Signing, Storers etc.
Understands v1 (no longer v1beta1). Wherever there was a use of pipeline resources, we convert to v1beta1 and call the v1beta1 logic.

In formats:

• v1: we convert to v1beta1 and analyze
• v2alpha1: we convert to v1beta1
• v2alpha2: convert to v1beta1
• v2alpha3: no need for conversion. Will look at v1, which is what the controller receives.

[chitrangpatel]

/assign @wlynch @lcarva PTAL 🙏 (specifically the storer and signing parts since I haven't used them a lot).

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	72.2%	2.0
pkg/chains/formats/slsa/extract/extract.go	61.5%	63.2%	1.6
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/artifact/v1beta1/append.go	Do not exist	100.0%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.6%	0.4
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/formats/slsa/v2alpha3/internal/external_parameters/external_parameters.go	Do not exist	87.5%
pkg/chains/formats/slsa/v2alpha3/internal/internal_parameters/internal_parameters.go	Do not exist	100.0%
pkg/chains/formats/slsa/v2alpha3/internal/pipelinerun/pipelinerun.go	Do not exist	83.8%
pkg/chains/formats/slsa/v2alpha3/internal/resolved_dependencies/resolved_dependencies.go	Do not exist	87.5%
pkg/chains/formats/slsa/v2alpha3/internal/taskrun/taskrun.go	Do not exist	86.5%
pkg/chains/formats/slsa/v2alpha3/slsav2.go	Do not exist	87.5%
pkg/chains/objects/objects.go	69.7%	31.5%	-38.2
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4

[chitrangpatel]

/assign @wlynch @lcarva

PTAL when you have a chance since this is a large PR.

Thanks 🙏

[chitrangpatel]

/approve

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	70.4%	0.2
pkg/chains/formats/slsa/extract/extract.go	61.5%	62.3%	0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.3%	0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/objects/objects.go	69.7%	32.7%	-37.0
pkg/reconciler/pipelinerun/controller.go	95.0%	86.4%	-8.6
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4
pkg/reconciler/taskrun/controller.go	94.1%	88.9%	-5.2

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	70.4%	0.2
pkg/chains/formats/slsa/extract/extract.go	61.5%	62.3%	0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.3%	0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/objects/objects.go	69.7%	32.7%	-37.0
pkg/reconciler/pipelinerun/controller.go	95.0%	86.4%	-8.6
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4
pkg/reconciler/taskrun/controller.go	94.1%	88.9%	-5.2

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	70.4%	0.2
pkg/chains/formats/slsa/extract/extract.go	61.5%	62.3%	0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.3%	0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/objects/objects.go	69.7%	32.7%	-37.0
pkg/chains/signing.go	73.7%	52.6%	-21.2
pkg/reconciler/pipelinerun/controller.go	95.0%	86.4%	-8.6
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4
pkg/reconciler/taskrun/controller.go	94.1%	88.9%	-5.2

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	70.4%	0.2
pkg/chains/formats/slsa/extract/extract.go	61.5%	62.3%	0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.3%	0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/objects/objects.go	69.7%	32.7%	-37.0
pkg/reconciler/pipelinerun/controller.go	95.0%	86.4%	-8.6
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4
pkg/reconciler/taskrun/controller.go	94.1%	88.9%	-5.2

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	70.4%	0.2
pkg/chains/formats/slsa/extract/extract.go	61.5%	62.3%	0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.3%	0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/objects/objects.go	69.7%	32.7%	-37.0
pkg/chains/storage/gcs/gcs.go	70.1%	64.2%	-5.9
pkg/reconciler/pipelinerun/controller.go	95.0%	86.4%	-8.6
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4
pkg/reconciler/taskrun/controller.go	94.1%	88.9%	-5.2

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	70.4%	0.2
pkg/chains/formats/slsa/extract/extract.go	61.5%	62.3%	0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.3%	0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/objects/objects.go	69.7%	32.7%	-37.0
pkg/chains/storage/gcs/gcs.go	70.1%	64.2%	-5.9
pkg/reconciler/pipelinerun/controller.go	95.0%	86.4%	-8.6
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4
pkg/reconciler/taskrun/controller.go	94.1%	88.9%	-5.2

[wlynch]

Overall looks good! Some final questions, otherwise should be able to get this merged.

[aaron-prindle]

Thanks for the reviews @wlynch and @lcarva! @wlynch I believe in the final review the only remaining work item was to update the godoc for gcs.go - StorePayload stating that the method was updated to use v1 (from v1beta1) and messages will reflect that. I have updated the godoc as such now:

// StorePayload implements the storage.Backend interface.  As of chains v0.20.0+,
// this method has been updated to use Tekton v1 objects (previously v1beta1) and
// it's error messages have been updated to reflect this.

With this change I believe this should be mergable IIUC. Thanks!

[tekton-robot]

The following is the coverage report on the affected files.
Say /test pull-tekton-chains-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/artifacts/signable.go	70.2%	70.4%	0.2
pkg/chains/formats/slsa/extract/extract.go	61.5%	62.3%	0.8
pkg/chains/formats/slsa/extract/v1beta1/extract.go	Do not exist	61.5%
pkg/chains/formats/slsa/internal/material/material.go	92.2%	92.3%	0.1
pkg/chains/formats/slsa/internal/material/v1beta1/material.go	Do not exist	92.2%
pkg/chains/formats/slsa/v1/intotoite6.go	88.9%	30.8%	-58.1
pkg/chains/formats/slsa/v2alpha1/slsav2.go	85.7%	50.0%	-35.7
pkg/chains/formats/slsa/v2alpha2/slsav2.go	87.5%	28.0%	-59.5
pkg/chains/objects/objects.go	69.7%	32.7%	-37.0
pkg/chains/storage/gcs/gcs.go	70.1%	64.2%	-5.9
pkg/reconciler/pipelinerun/controller.go	95.0%	86.4%	-8.6
pkg/reconciler/pipelinerun/pipelinerun.go	80.5%	82.9%	2.4
pkg/reconciler/taskrun/controller.go	94.1%	88.9%	-5.2

[wlynch]

/lgtm

[chitrangpatel]

/kind misc

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chitrangpatel, JeromeJu, wlynch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [chitrangpatel,wlynch]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment