Add basic DSSE equivalent for Metadata API and configurable DSSE support in ngclient

[lukpueh]

EDIT 2023/08/21: Rebased a.o. on top of #2361
EDIT 2023/10/12:

• Updated commit reference after rebase on Metadata API: add get_verification_result method #2481
• Updated link to demo gist
• Updated PR description to reflect latest state

EDIT 2024/02/21:

• Rebase
• Remove client example changes and demo

---

Fixes #2385

Notes for reviewer

• The largest chunk of the diff is a pure code move (503680a)
• The commit messages provide detailed information about changes, but not all changes end up in the final diff
• There is a demo included below

Background
The search for a common container abstraction for DSSE envelope and traditional Metadata has been unsuccessful in the past (see #2246, #2292). Some reasons are:

• Although, they look very similar, their usage patterns are different enough to make any common abstraction overly complex
• The life time of the container is too short to justify an overly complex solution.
• Even though application logic can be mostly agnostic to the container format. There are certain moments, where a deliberate decision for a container format should be made.

Summary of changes

• Adds basic DSSE equivalent for Metadata API called SimpleEnvelope API
• Make ngclient Updater configurable to deserialise TUF payloads either using Metadata API or SimpleEnvelope API

Demo (removed in b6fa05b)

# Install python-tuf with dsse support
git clone https://github.com/lukpueh/tuf -b add-dsse
cd tuf && pip install -r requirements/dev.txt

# Create and serve a simple static dsse demo repo
curl https://gist.githubusercontent.com/lukpueh/91224e528af5332ef9c779075cf71e06/raw/eb9e5dc1943710fff938674f0eb144b1b9a7edd4/serve_simple_dsse_repo.py \
    | python -

# Run example client in another terminal (same env and dir)
cd examples/client
./client tofu
./client download --use-dsse file1.txt

[lukpueh]

@jku: This is good for another round of review. The last 5 commits update the PR as we discussed (see commit messages for details). Remaining things I'd like to ticketize:

• Server-side tooling
  We should figure out how we want to support DSSE in our repo-side tooling, e.g. repository simulator (for tests) and repository abstraction (used in examples). Supporting both simultaneously, as ngclient does with this PR, might not be worth the effort. Also, if we find a solution for repo abstraction first, maybe we can use that in repo simulator.

• API (Docs)
  Currently, payload classes (Signed, Root, Targets), etc. are imported into both Metadata API and Envelope API, to preserve backwards compatibility of Metadata API and make both APIs fully independently usable. If we keep it that way, we need to figure out if we want to document those classes in both namespaces on readthedocs.

[jku]

I think this is good to go.

• I can't yet see how the repository side will work out with Envelope but I also don't think we will find out without merging this and then someone trying it out -- I suspect that unlike ngclient repositories are going to support Metadata or DSSE, not both.
• TrustedMetadataSet refactor is a good idea even without the rest of the PR
• Testing is lacking but I guess the first step towards improving that would be to write a RepositorySimulator for DSSE (or make the existing one support DSSE)

Thanks for the rewrites, I think it looks impressive:

$ git diff --stat origin/develop...lukpueh/add-dsse -- tuf/
...
2074 insertions(+), 1813 deletions(-)

[lukpueh]

Thanks for the rewrites, I think it looks impressive:

Thanks for the review! Your comments sound reasonable. Let me push another commit and ping for one last look.

$ git diff --stat origin/develop...lukpueh/add-dsse -- tuf/
...
2074 insertions(+), 1813 deletions(-)

Yes, sorry about that. I'm happy to move the initial commit, which is a pure code move, to a separate PR. Then it's a little less intimidating.

git diff --stat 503680a
...
 9 files changed, 564 insertions(+), 160 deletions(-)

[jku]

I pasted the stats as a positive thing: 250 added LOC for this is not bad :)

Sure, the _payload.py code move and especially the TrustedMetadataSet refactor could have been separate PRs but I think they're fine here.

[trishankatdatadog]

Looks like excellent work (as usual), thanks, Lukas!

Do you think it might be worthwhile to drop support for Canonical JSON as an envelope, and make it DSSE-only by default going forward? We might need a MAJOR release for that and some deprecation notice. Then again, shouldn't people be free to support whatever envelope they like (e.g., JOSE)?

[trishankatdatadog]

Suggested change

	METADATA = 1
	CJSON = 1

[trishankatdatadog]

Suggested change

	SIMPLE = 2
	DSSE = 2

[lukpueh]

Thanks for the comments, Trishank!

Do you think it might be worthwhile to drop support for Canonical JSON as an envelope, and make it DSSE-only by default going forward?

That would be nice, but I fear it would be extremely disruptive for any existing TUF user, if python-tuf (most notably ngclient) just stops supporting the cjson metadata wrapper.

Then again, shouldn't people be free to support whatever envelope they like (e.g., JOSE)?

I thinks so yes. Right now users can choose between "Envelope API" (dsse) or "Metadata API" (cjson) and configure the client to expect one or the other. I'm unsure if python-tuf can/should be changed to support arbitrary wrappers. I tried but couldn't find a good design for that.

[jku]

Then again, shouldn't people be free to support whatever envelope they like (e.g., JOSE)?

This has two really major disadvantages:

• each new option increases software complexity (which is bad for multiple reasons)
• each new option lowers chances of interoperability (an area where TUF has traditionally had significant issues)

These are downsides for DSSE as well, but DSSE balances that by improving other aspects of TUF (deserialization safety and ease of implementation).

The interop argument is also why dropping support for the TUF metadata format is unlikely to be a real option for a very long time.

[trishankatdatadog]

Agreed with both of you, and trust that there is no good common API for CJSON and DSSE (let alone other envelopes) in python-tuf right now. We can push down support for other envelopes to other implementations, or later if need be here.

[coveralls]

Pull Request Test Coverage Report for Build 8004729617

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• -10 of 800 (98.75%) changed or added relevant lines in 6 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.08%) to 97.725%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
tuf/ngclient/updater.py	15	16	93.75%
tuf/ngclient/_internal/trusted_metadata_set.py	72	74	97.3%
tuf/api/_payload.py	644	651	98.92%

Totals
Change from base Build 7991070707:	-0.08%
Covered Lines:	1463
Relevant Lines:	1488

---

💛 - Coveralls

[lukpueh]

@jku, I addressed your comments and rebased on a pretty different develop compared to before.

Here are two git command that might be helpful to re-review and make sure I didn't mess up while resolving conflicts

# Check if the diffs to develop before and after the rebase roughly match 
git range-diff 6fed68b..c96ec55 develop..add-dsse

# Check if all new metadata.py changes indeed moved to _payload.py
git diff develop:tuf/api/metadata.py tuf/api/_payload.py

[jku]

tuf-api.metadata.LengthOrHashMismatchError doesn't exist anymore but maybe that's fine

[lukpueh]

Good eyes! Can't remember, if I left it out on purpose or forgot. Let me re-export it, just to be safe.