Add --include-non-distributable flag to imgpkg copy

[ewrenn8]

Imgpkg should support a flag to allow the user to relocate images with foreign layers to airgapped envs.

Slack thread for some discussion: https://kubernetes.slack.com/archives/CH8KCCKA5/p1598013615011900

[pivotaljohn]

This is a rather critical feature for our users on Windows. Pulling in for this upcoming release.

[aaronshurley]

Consolidating #71 into this issue.

Story

As a operator wishing to collect a bundle that includes software that includes non-distributable bits (e.g. Windows-based products) for use in an air-gapped scenario
I want to be able to override the default behavior of skipping transferring so-called non-distributable layers ("foreign layers" in the docker manifests, "non-distributable" in OCI manifests)
and I want the bits of those layers to be pulled and included in the tarball of the bundle
and I want those bits to be loaded into the registry I copy the tarballed bundle.

Context

Original GitHub Issue: imgpkg#8

Requested by the .NET Build team (Malini Valliath, PM) multiple times since this summer.

PCS/MAPBU leadership has pointed to the importance of servicing our Windows-using customers — Ajay pointed out the importance of Windows support during an all-hands meeting, noting that a significant percentage of our customers use Windows.

Acceptance Criteria

Note: we say, "foreign layers", here. We wish for all this to be true for any non-distributable layer.

🟢 Copying foreign layers to a tarball

Given a bundle which includes a reference to an image containing a foreign layer (e.g. a bundle that references Microsoft's nanoserver — mcr.microsoft.com/windows/nanoserver:1809-amd64)
When I copy that bundle to a tarball
And I indicate that non-distributable layers (including foreign layers) should be included

$ imgpkg copy -b index.docker.io/k8slt/bundle-with-foreign-layers --to-tar the-football.tar --include-non-distributable

Then the tarball includes the contents of the layer

🟢 Copying a tarball containing one or more foreign layers to a registry

Given a tarball containing a bundle which includes one or more foreign layers
And a registry enabled to receive foreign layers (⁉️)
When I copy the bundle to an image reference (i.e. a location in a registry)
And I indicate that non-distributable layers (including foreign layers) should be included

$ imgpkg copy --from-tar the-football.tar --to-repo gcr.io/cf-k8s-lifecycle-tooling-klt/bundle-with-foreign-layers --include-non-distributable

Then I can find that bundle in the registry AND its digest has not changed
And I can find those foreign layers in the registry

🟡 Warn when foreign layer from registry is skipped

Given a bundle which includes a reference to an image containing a non-distributable layer (e.g. a foreign layer)
When I copy the bundle with no additional instruction to a tarball

$ imgpkg copy -b <image ref to bundle> --to-tar the-football.tar

Then I see a warning next to each non-distributable layer that indicates "skipped: foreign layer" or "skipped: non-distributable" (depending on the exact media type), similar to how Docker does it
And I see a warning in the summary that notes that one or more layers are not in the tarball because they are "non-distributable/foreign layers" and a link to documentation on how to include them, if desired
And I see that the overall operation is "Successful (with warnings)."
And the tarball does NOT include those layers (neither in the tarball itself, nor in the manifest.json)

🟡 Warn when foreign layer from tarball is skipped

Given a tarball containing a bundle which includes one or more foreign layers
When I copy the bundle to an image reference (i.e. a location in a registry), with no additional instruction

$ imgpkg copy --from-tar the-football.tar --to-repo gcr.io/cf-k8s-lifecycle-tooling-klt/bundle-with-foreign-layers

Then I see a warning next to each non-distributable layer that indicates "skipped: foreign layer" or "skipped: non-distributable" (depending on the exact media type), similar to how Docker does it
And I see a warning in the summary that notes that one or more layers are not in the tarball because they are "non-distributable/foreign layers" and a link to documentation on how to include them, if desired
And I see that the overall operation is "Successful (with caveats)."
And I can NOT find those foreign layers in the registry

📚 Documentation

Given I want to know more about how imgpkg handles non-distributed/foreign layers
When I read the documentation around the copy command
Then I discover/understand:

1. what are non-distributable layers:

• how they are identified in both OCI manifests and Docker manifests) and
• what purpose they serve
• (all this with links to authoritative sources, balancing clarity with minimizing duplicating information)

1. that by default imgpkg respects the non-distributable nature of foreign layers
2. that — as instructed by Microsoft — such layers can be copied
3. known limitations. 'foreign layers' don't appear to work on dockerhub. validate this is true and add any other registries that

---

Implementation Notes

• on Docker Hub, "side-loading" non-distributable layers and later referencing them in a manifest does not seem to work. See the comments in this story for details.

Helpful Resources/Notes

• It appears that the GGCR code base does not support writing an image that contains 'non-distributable' image layers (please verify this for yourself). Micah has provided a proof of concept that shows a 'workaround' https://github.com/micahyoung/registry-foreign-layers-testing.git
• Docker Registry API: Pushing an Image — nicely details the step-by-step of how to push an image.
• Windows Containers FAQ: How Do I Make my Container Images Available on air-gapped Machines?
• There are two image manifest specs, (oci and docker). Both have a way of marking a layer as non-distributable.
  • oci marks an image layer as non-distributable by using the media type: application/vnd.oci.image.layer.nondistributable.v1.taror application/vnd.oci.image.layer.nondistributable.v1.tar+gzip
  • docker marks an image layer as non-distributable by using the media type: application/vnd.docker.image.rootfs.foreign.diff.tar.gzip
• this will likely need to work with artifactory, could reach out to Micah to see what they used

[DennisDenuto]

Task list:

• Improve error message when the following occurs: copying a repo to a tar without the --include-non-distr flag, and then copying from the tar to a repo with the --include-non-distr flag should fail with a meaningful error message. currently it is failing with a cryptic error message
• recursively iterate through image and imageindexes when printing a warning message to the user. https://github.com/vmware-tanzu/carvel-imgpkg/blob/caaacb7495bc4a9c518ac71aa42f3252db04b7b1/pkg/imgpkg/cmd/copy_repo_src.go#L171

[aaronshurley]

This was released in v0.4.0.