Generate Location OCI Image

[joaopapereira]

Describe the problem/challenge you have
Scenario: When a Bundle is present in Registry registry.io and the images are in other.reg.io.
Basically, this scenario happens when a user does imgpkg push of a bundle but not execute imgpkg copy.

Given that other.reg.io needs credentials to download an Image
When executing imgpkg pull imgpkg tries to check if the Images are present in the same Repository as the Bundle and if not, it tries to check if the Images are present in the Location provided by the ImagesLock file. Because other.reg.io needs specific credentials it fails.

Describe the solution you'd like

After talking with @cppforlife we have a proposal to try to mitigate this problem.
In carvel-dev/carvel-community#22 we propose the creation of a Location OCI Image that will contain the location that the OCI Images were copied to. Our idea is to piggyback on this concept and split it from the proposal so that we can fix this problem.

How does this fix the problem?
When imgpkg pulls the Bundle it currently tries to find all the Images associated with the Bundle to see if they were copied or not. If we provide this Locations OCI Image we do not need to rely on checking all the Images to see if they are present since the Copy was done to the current Bundle location.

How imgpkg would do this?
When imgpkg copy is executed it will create an OCI Image that contains 1 layer with 1 file at the root called images-locations.yml that will have the following layout

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLocation
images:
  - image: world.io/img3@sha256:aaaaaaaaaa
    bundle: false

Field explanation:

• apiVersion (required; string) Version of this configuration.
• kind (required; ImagesLocation) Type of configuration, this value should always be ImagesLocation. Used to allow imgpkg to understand what type of configuration this document is defining
• images (required; []array) Must contain on entry per OCI Images present in the ImagesLock file.
• images[].image (required; string) This value MUST match an OCI Image defined in the ImagesLock file for the Bundle.
• images[].bundle (required; boolean) true is this image is a Bundle.

When copying a Bundle between Registries and/or Repositories this new OCI Image will be created in the destination Repository and will be tagged with the tag sha256-{Bundle SHA}.image-locations.imgpkg. This is not a perfect solution since Tags are mutable, but this will fix the problem for now.

Caveat 1
Since imgpkg would be using tags to find the new location for the OCI Images of a particular Bundle there is no guarantee that this tag is accurate. This should be considered the best effort to locate the OCI Images. When the tag is not present in the expected location imgpkg will assume that the copy was done using an older version of imgpkg.

Caveat 2
When replication is present, if it is set incorrectly, imgpkg cannot guarantee that all the OCI Images are present in the specified location. We assume that they are, if that is not the case an error will happen further down the line when trying to use these OCI Images

Searching order for OCI Image

1. When Locations OCI Image is present, the location will be assumed to be: Bundle Registry URL + repository field specified in the ImagesLocation configuration
2. When Locations OCI Image is NOT present, check if the Image exists in the location: Bundle Registry URL + Bundle Repository + @sha256:{Image SHA}
   2.1. If the OCI Image cannot be found in the prior Location, assumes (does not check) that the OCI Image Location is the one provided in the ImagesLock file

Anything else you would like to add:
Issue #129 will be addressed at the same time because in the new case we no longer need to check that all the Images are present before we change the ImagesLock file pulled.

Notes:

• We decided to use the above tag for the locations file to allow an easier creation of replication rules *.imgpkg
• We will be changing as part of a prior one the Tag that imgpkg adds automatically to all the OCI images from imgpkg-sha256-{Image SHA} to sha256-{image SHA}.imgpkg to accomplish the prior note

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help to work on this issue.

[joaopapereira]

Acceptance criteria:

Scenario 1: 🟢 Copy bundle only with image creates locations image
Given I create a bundle executing imgpkg push -b localhost:5000/new-bundle:bundle -f examples/basic-step-2
When I copy the bundle executing imgpkg copy -b localhost:5000/new-bundle:bundle --to-repo localhost:5000/new-location
Then I should see a success message
When I execute imgpkg pull -i localhost:5000/new-location:sha256-6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a.image-locations.imgpkg -o tmp
And I execute cat tmp/images-locations.yml
Then I should see

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLocation
images:
  - image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    repository: new-location
    bundle: false

---

Scenario 2: 🟢 Copy bundle with bundles creates one locations Image per bundle
Given I create a bundle executing imgpkg push -b localhost:5000/nested-bundle -f examples/basic-step-2
And I have the following ImagesLock file in example/basic-step-3/.imgpkg/images.yml

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLock
images:
- image: localhost:5000/nested-bundle@sha256:6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a

And I create a bundle executing imgpkg push -b localhost:5000/new-bundle:bundle -f examples/basic-step-3
When I copy the bundle executing imgpkg copy -b localhost:5000/new-bundle:bundle --to-repo localhost:5000/new-location
Then I should see a success message
When I execute imgpkg pull -i localhost:5000/new-location:sha256-25663a2243cb7ae1bfa904ba614a434437107eb8cd615f6285804a5e30023480.image-locations.imgpkg -o tmp
And I execute cat tmp/images-locations.yml
Then I should see

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLocation
images:
  - image: localhost:5000/nested-bundle@sha256:6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a
    bundle: true

When I execute imgpkg pull -i localhost:5000/new-location:sha256-6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a.image-locations.imgpkg -o tmp
And I execute cat tmp/images-locations.yml
Then I should see

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLocation
images:
  - image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    bundle: false

---

Scenario 3: 🟢 Copy bundle twice with image creates locations image pointing to the latter location
Given I create a bundle executing imgpkg push -b localhost:5000/new-bundle:bundle -f examples/basic-step-2
When I copy the bundle executing imgpkg copy -b localhost:5000/new-bundle:bundle --to-repo localhost:5000/new-location
Then I should see a success message
When I copy the bundle executing imgpkg copy -b localhost:5000/new-location:bundle --to-repo localhost:5000/yet-another-location
Then I should see a success message
When I execute imgpkg pull -i localhost:5000/yet-another-location:sha256-6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a.image-locations.imgpkg -o tmp
And I execute cat tmp/images-locations.yml
Then I should see

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLocation
images:
  - image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    bundle: false

---

Scenario 4: 🟢 Copy bundle image to repository with namespaces creates locations image pointing to the repository with namespaces
Given I create a bundle executing imgpkg push -b localhost:5000/new-bundle:bundle -f examples/basic-step-2
When I copy the bundle executing imgpkg copy -b localhost:5000/some/new-bundle:bundle --to-repo localhost:5000/some/namespace/new-location
Then I should see a success message
When I execute imgpkg pull -i localhost:5000/some/namespace/new-location:sha256-6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a.image-locations.imgpkg -o tmp
And I execute cat tmp/images-locations.yml
Then I should see

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLocation
images:
  - image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    bundle: false

---

Scenario 5: 🟢 Copy bundle twice first with an older version of imgpkg creates locations image on the second copy
Given I create a bundle executing imgpkg push -b localhost:5000/new-bundle:bundle -f examples/basic-step-2
When I copy the bundle using an older version of imgpkg executing imgpkg copy -b localhost:5000/new-bundle:bundle --to-repo localhost:5000/old-imgpkg-img
Then I should see a success message
When I execute imgpkg pull -i localhost:5000/old-imgpkg-img:sha256-6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a.image-locations.imgpkg -o tmp
Then I should see an error message manifest unknown
When I copy the bundle executing imgpkg copy -b localhost:5000/old-imgpkg-img:bundle --to-repo localhost:5000/yet-another-location
Then I should see a success message
When I execute imgpkg pull -i localhost:5000/yet-another-location:sha256-6716afd7a68262a37d3f67681ed9dedf3b882938ad777f268f44d68894531f7a.image-locations.imgpkg -o tmp
And I execute cat tmp/images-locations.yml
Then I should see

apiVersion: imgpkg.carvel.dev/v1alpha1
kind: ImagesLocation
images:
  - image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0
    bundle: false

---

Notes:

• If the copy fails for any reason we should ensure that the locations OCI Image is not created
• Each location's image will only contain information about the ImagesLock for that particular bundle. This means that when we change pull we will still need to go to each Bundle to find if the Bundle has a location image or not.

[cppforlife]

introduced in https://github.com/vmware-tanzu/carvel-imgpkg/releases/tag/v0.9.0