Add rekor test harness to presubmit tests

[priyawadhwa]

This will test critical tests against the last three versions of rekor released.

I picked out a subset of the current e2e tests which I think are important to run against previous versions.

addresses https://github.com/sigstore/public-good-instance/issues/417

Signed-off-by: Priya Wadhwa [email protected]

Summary

Release Note

Documentation

[codecov-commenter]

Codecov Report

Merging #921 (603cc15) into main (f2d3630) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #921      +/-   ##
==========================================
- Coverage   48.25%   48.20%   -0.06%     
==========================================
  Files          62       62              
  Lines        5398     5398              
==========================================
- Hits         2605     2602       -3     
- Misses       2511     2513       +2     
- Partials      282      283       +1     

Impacted Files	Coverage Δ
pkg/types/alpine/v0.0.1/entry.go	55.46% <0.00%> (-1.27%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us.

[bobcallaway]

IIUC, this uses the source at head to compile a new rekor-cli that is then used against prior versions of the server? I'm not sure how frequent of a situation that will be.

I was expecting to see something more similar to this:

• Define a queue of versions to test (similar to how you've picked the last 3 versions here). Call that sorted (ascending) queue Q
• Define a set of versions you have tested, let's call that S, initially empty
• Start by popping a version off of the queue Q and adding it to the set S
• let v=max(S)
• start the server with docker-compose up rekor-server and dependencies (redis, mysql, trillian) for version v,
• for all values in S, run HARNESS_TESTS using the rekor-cli version v (we should also capture entryIDs for all inserted entries, and ensure all are readable with rekor-cli versions in S)
• Pop the next version off of the queue Q and add to S
• Repeat this process, assuming the database and trillian instances are persisted across rekor-server version upgrades (similar to what was done in sharding). This is critical, since it mirrors what will happen in prod.

[bobcallaway]

Suggested change

	# Copyright 2021 The Sigstore Authors.
	# Copyright 2022 The Sigstore Authors.

[priyawadhwa]

@bobcallaway so I set up something more like what you were describing:

• take the last three versions (example v1, v2, v3)
• start by setting up the server at v1, & test against all 3 versions of the CLI
• when upgrading the server for v2 and v3, restart just rekor-server and maintain the same mysql/trillian backend. test each against all 3 versions of the CLI as well.

does that seem reasonable as a skeleton?

[priyawadhwa]

update: i have tests which should more closely mimic updates in prod. for every server version upgrade, they'll add new entries and also make sure they can still get all the old entries.

i might add that in a follow-up pr so this one doesn't get too big

[bobcallaway]

For inserts at version X and testing if we can run the exact same insert at version X+1, I think this does it. As you note this doesn't cover reading back previously inserted values inserted under code versions that were backlevel from the proposed change.

[priyawadhwa]

Thanks @bobcallaway! Addressed comments, and I'll open a follow-up PR once this is in to check reading previously inserted values.