inclusion proof returned a tree size larger than the verified tree size

[ddworken]

I'm using slsa-verifier as a library in my project to do secure updates. Recently, updates have started failing with the output:

Getting rekor entry error error verifying tlog entry: inclusion proof returned a tree size larger than the verified tree size: error verifying root hash, trying Redis search index to find entries by subject digest
could not find a matching valid signature entry: got unexpected errors expected matching UUID: verifying tlog entry 362f8ecba72f4326212f6d724cfef27ba534207bff159c6416bac927c2c024c2cfdb923d86ad8f8a

The first line appears to be a log message, the second line is the error returned by verifiers.Verify. You can see my verification code here and my releasers here.

This update flow used to work, but it seems as though it was recently broken. Do you happen to have any advice on how to debug/fix this?

[asraa]

Hey @ddworken! Thanks for the report, I'll investigate.

We had started seeing this problem in our e2e tests, due to a race condition in Rekor: sigstore/rekor#1055, but we thought the fix was rolled out in production as of last tuesday.

I'll try to see if I can reproduce the problem locally.

It should be transient since it's a race condition, so I would attempt to try restarting. I'll post any updates as I dig in.

[ddworken]

Ah interesting! It does seem like that is what I ran into. To reproduce the error I originally reported:

wget https://github.com/ddworken/hishtory/releases/download/v0.128/hishtory-linux-amd64
wget https://github.com/ddworken/hishtory/releases/download/v0.128/hishtory-linux-amd64.intoto.jsonl
slsa-verifier -artifact-path hishtory-linux-amd64 -provenance hishtory-linux-amd64.intoto.jsonl -source github.com/ddworken/hishtory

which gives:

Getting rekor entry error error verifying tlog entry: inclusion proof returned a tree size larger than the verified tree size: error verifying root hash, trying Redis search index to find entries by subject digest
FAILED: SLSA verification failed: could not find a matching valid signature entry: got unexpected errors expected matching UUID: verifying tlog entry 362f8ecba72f43260d72dafbe4afe657b6106a542c5c22aa2803233a9f61a793563baf4985aa564f

But I seem to now be running into a separate bug. To reproduce this one:

wget https://github.com/ddworken/hishtory/releases/download/v0.133/hishtory-linux-amd64
wget https://github.com/ddworken/hishtory/releases/download/v0.133/hishtory-linux-amd64.intoto.jsonl
slsa-verifier -artifact-path hishtory-linux-amd64 -provenance hishtory-linux-amd64.intoto.jsonl -source github.com/ddworken/hishtory

Which gives:

Getting rekor entry error error verifying tlog entry: leafHash has unexpected size 40, want 32: verifying inclusion proof, trying Redis search index to find entries by subject digest
FAILED: SLSA verification failed: could not find a matching valid signature entry: got unexpected errors expected matching UUID: verifying tlog entry 24296fb24b8ad77a2450ad5fe4415aa44afe94587a29399b0c2e319997c7437923fcc56d2cf4b1a7

And this is the github action run that generated these files.

[asraa]

Nice!

The second separate bug you ran into is because of issues we're currently fixing. The prod rollout last week also changed the return response of Rekor to return back full UUIDs (since 40, rather than 32): so thees fixes are on the way.

I just finished releasing slsa-verifier 1.3.1 and 1.2.1: that should include the fix for the separate bug. I'm working on the backports for v1.1/v1.0.

I'm still really interested in if the first error is occurring after the rollout...

[asraa]

Could you try if bumping to https://github.com/slsa-framework/slsa-verifier/releases/tag/v1.3.1 fixes the second issue?

[ddworken]

Ah yes, it appears bumping to v1.3.1 fixed the second issue! I also checked and I can no longer produce the original issue: all newly created attestations appear to be valid.

Thank you very much for the help and quick replies!