verify-blob not finding tlog on rekor, but it is there

[caarlos0]

Description

I might have found a bug, not sure if here, rekor, or elsewhere... or maybe my understanding of things is a bit wrong... in any case, here it goes:

First thing I noticed is that rekor-cli does not find any entries for some files:

$ rekor-cli search --artifact https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt
no matching entries found

$ rekor-cli search --artifact https://github.com/goreleaser/goreleaser/releases/download/v1.4.0/checksums.txt
no matching entries found

$ rekor-cli search --artifact https://github.com/goreleaser/goreleaser/releases/download/v1.3.0/checksums.txt
no matching entries found

$ rekor-cli search --artifact https://github.com/goreleaser/nfpm/releases/download/v2.12.1/checksums.txt
Found matching entries (listed by UUID):
686fb1ee555cbc47416a99b8e8797cfe07b45946ed03df0908bcd8e4a307c78d

$ rekor-cli search --artifact https://github.com/goreleaser/nfpm/releases/download/v2.11.0/checksums.txt
no matching entries found

$ rekor-cli search --artifact https://github.com/goreleaser/nfpm/releases/download/v2.11.3/checksums.txt
no matching entries found

Here, it seems like after some time they are not available anymore, or at least, not searchable?

So, if I try to verify without the PEM, it doesn't work:

$ COSIGN_EXPERIMENTAL=1 cosign verify-blob \
        --signature https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt.sig \
       https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt
Error: verifying blob [https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt]: could not find a tlog entry for provided blob
main.go:46: error during command execution: verifying blob [https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt]: could not find a tlog entry for provided blob

But, if I verify it with the PEM as well, it prints an UUID:

$ COSIGN_EXPERIMENTAL=1 cosign verify-blob \
        --cert https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt.pem \
        --signature https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt.sig \
       https://github.com/goreleaser/goreleaser/releases/download/v1.4.1/checksums.txt
tlog entry verified with uuid: "190dbbf9ffc81cc77508cbdf3026a9acac00a879a70da8e67642d91e5221c063" index: 1173394
Verified OK

and then, if I get that UUID from rekor:

$ rekor-cli get --uuid 190dbbf9ffc81cc77508cbdf3026a9acac00a879a70da8e67642d91e5221c063
LogID: c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d
Index: 1173394
IntegratedTime: 2022-01-27T04:16:40Z
UUID: 190dbbf9ffc81cc77508cbdf3026a9acac00a879a70da8e67642d91e5221c063
Body: {
  "HashedRekordObj": {
    "data": {
      "hash": {
        "algorithm": "sha256",
        "value": "13555fbe97a8c10b7cc46ff659c80cb8cb3d51d37aa5f1daa9456b3ebb4995a9"
      }
    },
    "signature": {
      "content": "MEUCID/ZYTmS62dJWlnr13cZ+FJBlKhSBxHMdaCiH8KjmN71AiEAj6YqkN5L61yhwjD8bhykBCQwPm6f3v5twDlqgS9tjlA=",
      "publicKey": {
        "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHakNDQXFDZ0F3SUJBZ0lUU1ZnVXpxZ1BscGt5dng5dHpZQ21scHpaSWpBS0JnZ3Foa2pPUFFRREF6QXEKTVJVd0V3WURWUVFLRXd4emFXZHpkRzl5WlM1a1pYWXhFVEFQQmdOVkJBTVRDSE5wWjNOMGIzSmxNQjRYRFRJeQpNREV5TnpBME1UWXpPVm9YRFRJeU1ERXlOekEwTWpZek9Gb3dFekVSTUE4R0ExVUVDaE1JYzJsbmMzUnZjbVV3CldUQVRCZ2NxaGtqT1BRSUJCZ2dxaGtqT1BRTUJCd05DQUFTZEsrWEpmNlloYTdkc2wwNElqNEs3MXVYcVVySHEKYWRpS3JWZ1VGc0J4Zjcxc1lCNzcxaXJsWXgzZWFsajlQSStEL2V6UGNNRXdBdU8yblFLdlBzWjlvNElCdWpDQwpBYll3RGdZRFZSMFBBUUgvQkFRREFnZUFNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01ETUF3R0ExVWRFd0VCCi93UUNNQUF3SFFZRFZSME9CQllFRk0xcitFbnVUemd5dktWRjV3eGFKYVFYdndJa01COEdBMVVkSXdRWU1CYUEKRkZqQUhsK1JSYVZtcVhyTWtLR1RJdEFxeGNYNk1HQUdBMVVkRVFSWk1GZUdWV2gwZEhCek9pOHZaMmwwYUhWaQpMbU52YlM5bmIzSmxiR1ZoYzJWeUwyZHZjbVZzWldGelpYSXZMbWRwZEdoMVlpOTNiM0pyWm14dmQzTXZZblZwCmJHUXVlVzFzUUhKbFpuTXZkR0ZuY3k5Mk1TNDBMakV3RWdZS0t3WUJCQUdEdnpBQkFnUUVjSFZ6YURBVEJnb3IKQmdFRUFZTy9NQUVFQkFWaWRXbHNaREE1QmdvckJnRUVBWU8vTUFFQkJDdG9kSFJ3Y3pvdkwzUnZhMlZ1TG1GagpkR2x2Ym5NdVoybDBhSFZpZFhObGNtTnZiblJsYm5RdVkyOXRNRFlHQ2lzR0FRUUJnNzh3QVFNRUtHRXhORFEzCllUTTJNelUzT1RNMk5XWTBPRGswTlRoaFpEYzJNelptWkRBNE9HRTFZalkyWVdJd0hnWUtLd1lCQkFHRHZ6QUIKQmdRUWNtVm1jeTkwWVdkekwzWXhMalF1TVRBakJnb3JCZ0VFQVlPL01BRUZCQlZuYjNKbGJHVmhjMlZ5TDJkdgpjbVZzWldGelpYSXdDZ1lJS29aSXpqMEVBd01EYUFBd1pRSXhBTTZyVDEzTGVnZk9ZOGpvUDVuZWR5cFV4WnFlCjd3VFhXaVdmRnZpTFRCWmJkNU9pUERzdW5zZzdSVDFiZllQOGx3SXdTZmdOaGs1Q2hUWnpxREdPOVZMQllKaS8KZnBTRDBlcTBvaEt4S2szTmp1QkJZcE1nVXpadUVSVXgvRDhjR25rMgotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
      }
    }
  }
}

Similarly, rekor-cli verify --uuid 190dbbf9ffc81cc77508cbdf3026a9acac00a879a70da8e67642d91e5221c063 also works.

I also noticed that it doesn't print relevant info from GitHub's UIDC, e.g. the action run url and actor anymore... I have the impression that this did work at some point... maybe I'm misremembering thing though.

In any case, I would like to understand if this is expected, and if so, why - and how should I instruct users to verify the actual public key used to verify the release...

refs goreleaser/goreleaser#2876

[bobcallaway]

rekor-cli search queries an index that maintains a mapping from various keys to a specific entry in the log. it's unlikely but entirely possible that the index becomes out of sync with the state of the transparency log, so we'll need to investigate further what's going on here.

Both rekor-cli verify and cosign verify do a lookup directly from the transparency log by recreating the expected entry and seeing if it exists. That explains why when you omit the public key PEM you see a failure (there's not enough to fully recreate the entry), but if you query by UUID things work.

I also noticed that it doesn't print relevant info from GitHub's UIDC, e.g. the action run url and actor anymore... I have the impression that this did work at some point... maybe I'm misremembering thing though.

That information is stored in the code signing cert, which can be printed using openssl + some jq trickery:

bcallaway@bcallaway01:~/git/sigstore/rekor$ rekor-cli get --uuid 190dbbf9ffc81cc77508cbdf3026a9acac00a879a70da8e67642d91e5221c063 --format=json|jq -r .Body.HashedRekordObj.signature.publicKey.content|base64 -d | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            49:58:14:ce:a8:0f:96:99:32:bf:1f:6d:cd:80:a6:96:9c:d9:22
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: O = sigstore.dev, CN = sigstore
        Validity
            Not Before: Jan 27 04:16:39 2022 GMT
            Not After : Jan 27 04:26:38 2022 GMT
        Subject: O = sigstore
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:9d:2b:e5:c9:7f:a6:21:6b:b7:6c:97:4e:08:8f:
                    82:bb:d6:e5:ea:52:b1:ea:69:d8:8a:ad:58:14:16:
                    c0:71:7f:bd:6c:60:1e:fb:d6:2a:e5:63:1d:de:6a:
                    58:fd:3c:8f:83:fd:ec:cf:70:c1:30:02:e3:b6:9d:
                    02:af:3e:c6:7d
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: 
                Code Signing
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                CD:6B:F8:49:EE:4F:38:32:BC:A5:45:E7:0C:5A:25:A4:17:BF:02:24
            X509v3 Authority Key Identifier: 
                keyid:58:C0:1E:5F:91:45:A5:66:A9:7A:CC:90:A1:93:22:D0:2A:C5:C5:FA

            X509v3 Subject Alternative Name: 
                URI:https://github.com/goreleaser/goreleaser/.github/workflows/build.yml@refs/tags/v1.4.1
            1.3.6.1.4.1.57264.1.2: 
                push
            1.3.6.1.4.1.57264.1.4: 
                build
            1.3.6.1.4.1.57264.1.1: 
                https://token.actions.githubusercontent.com
            1.3.6.1.4.1.57264.1.3: 
                a1447a363579365f489458ad7636fd088a5b66ab
            1.3.6.1.4.1.57264.1.6: 
                refs/tags/v1.4.1
            1.3.6.1.4.1.57264.1.5: 
                goreleaser/goreleaser
    Signature Algorithm: ecdsa-with-SHA384
         30:65:02:31:00:ce:ab:4f:5d:cb:7a:07:ce:63:c8:e8:3f:99:
         de:77:2a:54:c5:9a:9e:ef:04:d7:5a:25:9f:16:f8:8b:4c:16:
         5b:77:93:a2:3c:3b:2e:9e:c8:3b:45:3d:5b:7d:83:fc:97:02:
         30:49:f8:0d:86:4e:42:85:36:73:a8:31:8e:f5:52:c1:60:98:
         bf:7e:94:83:d1:ea:b4:a2:12:b1:2a:4d:cd:8e:e0:41:62:93:
         20:53:36:6e:11:15:31:fc:3f:1c:1a:79:36

[caarlos0]

Both rekor-cli verify and cosign verify do a lookup directly from the transparency log by recreating the expected entry and seeing if it exists. That explains why when you omit the public key PEM you see a failure (there's not enough to fully recreate the entry), but if you query by UUID things work.

ok, so my mental model was more or less accurate, thanks!

That information is stored in the code signing cert, which can be printed using openssl + some jq trickery:

ah, neat!

---

Another question: what's the recommended way to instruct users to check if the public key et al are what they actually say they are?

[cardil]

The recommended way should only use common tools, already preinstalled on most systems like base64, curl, openssl, maybe even jq.

Using rektor-cli is for this isn't the best way, as how one could get rektor-cli installed in safe way in the first place.

[caarlos0]

That would probably be ideal @cardil

@dlorenc are there any options for this?

I could use something like

cat checksums.txt.pem | 
  base64 -d | 
  openssl x509 -text -noout

to get some info about the certificate, but how do I verify everything without installing cosign, rekor-cli, etc? is it even possible?

[asraa]

Hi! Just FYI I think this is fixed, because we now also fallback on searching the redis index for the correct entry when you omit the PEM..

This works

COSIGN_EXPERIMENTAL=1 ./cosign verify-blob         --signature https://github.com/goreleaser/goreleaser/releases/download/v1.8.3/checksums.txt.sig        https://github.com/goreleaser/goreleaser/releases/download/v1.8.3/checksums.txt
tlog entry verified with uuid: 615e5aa4b6697c34a91c2350332a2f62c07c2fa0369439ae84fc614d9e46636c index: 2050842
Verified OK

[asraa]

RE the verifying public key, you could pin the verification against the workflow that signed, e.g. issuer github and subject name https://github.com/goreleaser/goreleaser/.github/workflows/build.yml@refs/tags/v1.8.3 but using the --cert-email flags seems to only check for email SAN, not the job workflow ref URI in the cert unfortunately.

[caarlos0]

happening again with some newly signed releases:

COSIGN_EXPERIMENTAL=1 cosign verify-blob \
  --signature https://github.com/goreleaser/goreleaser/releases/download/v1.10.3/checksums.txt.sig \
  https://github.com/goreleaser/goreleaser/releases/download/v1.10.3/checksums.txt
WARNING: No valid entries were found in rekor to verify this blob.

Transparency log support for blobs is experimental, and occasionally an entry isn't found even if one exists.

We recommend requesting the certificate/signature from the original signer of this blob and manually verifying with cosign verify-blob --cert [cert] --signature [signature].
Error: verifying blob [https://github.com/goreleaser/goreleaser/releases/download/v1.10.3/checksums.txt]: could not find a valid tlog entry for provided blob, found 1 invalid entries
main.go:62: error during command execution: verifying blob [https://github.com/goreleaser/goreleaser/releases/download/v1.10.3/checksums.txt]: could not find a valid tlog entry for provided blob, found 1 invalid entries

[bobcallaway]

looks like sigstore/rekor#891

[asraa]

actually, i did some debugging. it's a UUID issue. Fixing now.

[asraa]

EDIT: I never submitted this PR: #2058
This will fix it! Updating.

[caarlos0]

ah, thanks @asraa !