Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Signing images by digest that don't exist yet #1905

Closed
imjasonh opened this issue May 20, 2022 · 12 comments · Fixed by #2959
Closed

Signing images by digest that don't exist yet #1905

imjasonh opened this issue May 20, 2022 · 12 comments · Fixed by #2959
Labels
enhancement New feature or request good first issue Good for newcomers oci Integration between Sigstore and OCI registries/images

Comments

@imjasonh
Copy link
Member

Description

It should be possible to sign an image specified by digest that doesn't (yet) exist in the registry, so long as the user has permission to push to the repository. As an example, I should be able to cosign sign registry.biz/myuser/image@sha256:abcdef even if no image with that digest exists yet.

Signing registry.biz/myuser/image:tag should still require that that tag exists, because cosign needs to be able to lookup the digest the tag points to. The registry.biz/myuser repository also must exist, and be accessible to the user, because that's where we'll push the signature.

Pre-signing images because an image build tool can write a signature to the repository before the image being signed is pushed, for example if I enforce a policy that only signed images can exist in my registry. Signing after pushing introduces a race condition, especially if signing fails for some reason (e.g., I close the OIDC browser tab and walk away), leading to unsigned images.

This used to work in a previous version of cosign, but seems to have regressed (possibly #1616 @puerco?). If we decide to fix this, we should introduce a test that guards against future regressions. Re #1616 we should also make sure that you can sign the same non-existent digest multiple times successfully.

Currently:

$ COSIGN_EXPERIMENTAL=1 cosign sign gcr.io/imjasonh/blah@sha256:$(head -n 10 /dev/urandom | sha256sum | cut -d' ' -f1)
Generating ephemeral keys...
Retrieving signed certificate...
Your browser will now be opened to:
https://oauth2.sigstore.dev/auth/auth?access_type=online&client_id=sigstore&code_challenge=REDACTED&code_challenge_method=S256&nonce=REDACTED&redirect_uri=http%3A%2F%2Flocalhost%3A58857%2Fauth%2Fcallback&response_type=code&scope=openid+email&state=REDACTED
Successfully verified SCT...
Error: signing [gcr.io/imjasonh/blah@sha256:7145ec233c4ebd21780be57aef96e3e18894d92a00755c3fd8974f5e82c244c8]: accessing image: entity not found in registry
main.go:52: error during command execution: signing [gcr.io/imjasonh/blah@sha256:7145ec233c4ebd21780be57aef96e3e18894d92a00755c3fd8974f5e82c244c8]: accessing image: entity not found in registry
@imjasonh imjasonh added the enhancement New feature or request label May 20, 2022
@thesayyn
Copy link

Pre-signing images because an image build tool can write a signature to the repository before the image being signed is pushed, for example, if I enforce a policy that only signed images can exist in my registry. Signing after pushing introduces a race condition, especially if signing fails for some reason (e.g., I close the OIDC browser tab and walk away), leading to unsigned images.

I have this exact use case. I believe not having the signature available for a brief moment is as bad as not having the signature at all since this time period could overlap with the pod scheduling thus leading to the unsigned images being used by systems unless you have something in place to prevent this. This is okay if you control both the image itself and downstream systems. however, this is not easy if you don't control the downstream dependents.

@dlorenc
Copy link
Member

dlorenc commented Aug 29, 2022

We should definitely fix this.

@thesayyn
Copy link

We should definitely fix this.

I'd be willing to fix it if I am given the proper instructions. I am not familiar with the codebase at all but already familiar with gccr.

@znewman01 znewman01 added the good first issue Good for newcomers label Sep 2, 2022
@znewman01
Copy link
Contributor

FYI @imjasonh is probably a good person to give some guidance here; he's OOO this week but will be back next week. Give him a few days to catch up and then maybe ping this issue.

@imjasonh
Copy link
Member Author

imjasonh commented Sep 6, 2022

@puerco says #1616 unintentionally removed this ability, and we should add it back. AIUI this means using ociempty.SignedImage if the image-by-digest doesn't exist in the registry, determined using pkg/v1/remote.Head presumably.

If we're concerned about the race that introduces, we can do something smarter and more involved with ociremote.SignedEntity to have it return an ociempty.SignedEntity if it's not found, instead of failing.

@thesayyn if you're interested that would be great. The APIs should all feel very familiar coming from ggcr 😄

@thesayyn
Copy link

@imjasonh Definitely. I'll start by understanding what each of these APIs does. Thanks for the pointers.

@thesayyn
Copy link

thesayyn commented Sep 26, 2022

@imjasonh I have some questions here;

1- What if the non-existent-image-by-digest is an image index rather than an image (-r is no-op here)? is there a way to determine this?
2- What should be the behavior in case of a race condition? abort or overwrite?
3- If the behavior is to abort, how do we determine it? If-Match?

@znewman01 znewman01 added the oci Integration between Sigstore and OCI registries/images label Nov 22, 2022
@saisatishkarra
Copy link

saisatishkarra commented Jan 20, 2023

I had the same issue and wanted to summarize the discussion we had with sigstore slack channel:

  1. Not being able to sign images by digest before pushing to the registry defeats use cases such as the requirement to only have signed images in the registry
  2. I was told that this was possible with a few clients such as ko and not docker (not sure of the details) and has something with the way digest calculation and fetching takes place. This also makes it hard especially with docker/build-push-action that emits invalid/lazy generated digest to be propagated across different CI nodes when using OCI TAR outputs for sharing image artifacts for scan, sign, and attestation of assets first without pushing to the registry.

Workarounds suggested:

  1. Set up local docker registry proxy / remote staging registry to push image:tag -> fetch the digest -> run scan +sign -> publish to prod and discard the local proxy registry in CI.
  2. In case there is no proxy/staging registry, use unique / dummy tags (not like someone immediately can know about it) and then refer to the digest to perform signing. Con: pollutes registry

thanks @mattmoor and @jonjohnsonjr for clarification. I would really like to contribute to documenting these limitations if someone can point me around and get into specifics of the differences in the digest generation by clients that make this hard to achieve.

@gosixtyfive
Copy link

@imjasonh Steve here from the ko community meeting yesterday. Turns out you filed the ticket that I was looking at. Interested in working on this or at least understanding it. @thesayyn Are you still interested in working this one and #2603 that you also filed which is related?

@thesayyn
Copy link

I haven't been able to figure out what the behavior should be. I am still interested but got questions.

paolomainardi added a commit to paolomainardi/cosign that referenced this issue Mar 20, 2023
@paolomainardi
Copy link
Contributor

Just tried to open a PR to address this: #2824

mattmoor added a commit to mattmoor/cosign that referenced this issue May 7, 2023
🎁 This feature allows `cosign` to sign a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: sigstore#1905

/kind feature
mattmoor added a commit to mattmoor/cosign that referenced this issue May 7, 2023
🎁 This feature allows `cosign` to sign a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: sigstore#1905

/kind feature

Signed-off-by: Matt Moore <[email protected]>
mattmoor added a commit to mattmoor/cosign that referenced this issue May 7, 2023
🎁 This feature allows `cosign` to sign a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: sigstore#1905

/kind feature

Signed-off-by: Matt Moore <[email protected]>
mattmoor added a commit to mattmoor/cosign that referenced this issue May 7, 2023
🎁 This feature allows `cosign` to sign and attest a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: sigstore#1905

/kind feature

Signed-off-by: Matt Moore <[email protected]>
@mattmoor
Copy link
Member

mattmoor commented May 7, 2023

stale-bot got the two attempts at this. I took a crack at this blending the approaches, fixing cosign attest too, and adding some e2e test coverage 🤞

mattmoor added a commit to mattmoor/cosign that referenced this issue May 8, 2023
🎁 This feature allows `cosign` to sign and attest a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: sigstore#1905

/kind feature

Signed-off-by: Matt Moore <[email protected]>
mattmoor added a commit to mattmoor/cosign that referenced this issue May 8, 2023
🎁 This feature allows `cosign` to sign and attest a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: sigstore#1905

/kind feature

Signed-off-by: Matt Moore <[email protected]>
mattmoor added a commit to mattmoor/cosign that referenced this issue May 8, 2023
🎁 This feature allows `cosign` to sign and attest a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: sigstore#1905

/kind feature

Signed-off-by: Matt Moore <[email protected]>
priyawadhwa pushed a commit that referenced this issue Jun 2, 2023
* Feature: Allow cosign to sign/attest digests before they are uploaded.

:gift: This feature allows `cosign` to sign and attest a digest that doesn't exist yet, to support scenarios such as signing an image prior to pushing it.

This adapts the ideas from the two prior approaches, which were closed by stale-bot.

Fixes: #1905

/kind feature

Signed-off-by: Matt Moore <[email protected]>

* Apply suggestions from code review

Co-authored-by: Jon Johnson <[email protected]>
Signed-off-by: Matt Moore <[email protected]>

---------

Signed-off-by: Matt Moore <[email protected]>
Signed-off-by: Matt Moore <[email protected]>
Co-authored-by: Jon Johnson <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers oci Integration between Sigstore and OCI registries/images
Projects
None yet
8 participants