Create a bundle or image directly as a tar file

[joaopapereira]

Describe the problem/challenge you have
In a scenario where the artifact that will be shared is a tar, there might not be a need to use the registry as an intermediary step. This would allow the users to combine the commands imgpkg push + imgpkg copy --to-tar in a single command and without storing extra images in a registry

Describe the solution you'd like

Scenario for images
Given a set of files in examples/basic-step-1
When execute imgpkg build --tag v0.5.0 -f examples/basic-step-1 --to-tar some.tar
Then copy the image to a registryimgpkg copy --tar some.tar --to-repo localhost:5000/some-image
And retrieve it using imgpkg pull -i localhost:5000/some-image:v0.5.0

Scenario for bundles
Given a bundle in examples/basic-step-2
When execute imgpkg build --tag v0.5.0 -f examples/basic-step-1 --to-tar some-bundle.tar
Then copy the bundle to a registryimgpkg copy --tar some-bundle.tar --to-repo localhost:5000/some-bundle
And retrieve it using imgpkg pull -b localhost:5000/some-image:v0.5.0

Notes:

• Tags used during the build step should be kept when copying

Based on a request from Zihan Jiang: Slack conversations

[StevenLocke]

This feels like a good enhancement for imgpkg. One question we'd like to pose is whether this fits into a top level command or should be separated?
i.e. Could this be a subset of copy or should we extract it to a build command?

At the moment, my understanding is that tars do not have any tagging metadata. If that's a firm requirement, we may want to add that functionality to all of our tar creating.
We have an old desired feature to support a --to-tag flag, maybe that would be sufficient for the "note" behaviour.

[joaopapereira]

i.e. Could this be a subset of copy or should we extract it to a build command?

The idea behind doing this as a separated command is to separate these 2 concerns and also do not overload the copy command. Sounds more comprehensive to build an OCI Image tar from a set of files than copy a set of files into an OCI Image tar. But we can try to talk with some users to better understand if this is the best UX option.

At the moment, my understanding is that tars do not have any tagging metadata. If that's a firm requirement, we may want to add that functionality to all of our tar creating.
We have an old desired feature to support a --to-tag flag, maybe that would be sufficient for the "note" behaviour.

Currently when you do the flow push bundle to registry -> copy to tar -> push tar to registry we already preserve the original tag that we pushed the bundle with. (Check requirement to add some functionality on image push in #54.)

If you check https://github.com/vmware-tanzu/carvel-imgpkg/blob/develop/pkg/imgpkg/imagetar/tar_writer.go#L63 you will see that we already save and read a manifest.json file. That file contains the struct defined by https://github.com/vmware-tanzu/carvel-imgpkg/blob/53dbdc082f8135c4e5a5241fd751cb07f8edf247/pkg/imgpkg/imagedesc/types.go#L38
This struct contains tag information for all the images.

[danielhelfand]

@joaopapereira do we have enough information here to add this to our backlog?

@ZihanJiang96 would you have an interest in submitting a pr for this?

[joaopapereira]

Yes, I think we have enough information for this implementation. Nevertheless, we should do some UX investigation to ensure that our users understand what the command does.

[ZihanJiang96]

Current want I am doing to get a img tar:

-> imgpkg push -i $IMAGE_NAME:$TAG -f $PATH_TO_FILE
-> docker save $IMAGE_NAME:$TAG -o example.tar

My expectation is something like this, in this case, we never push the image to the registry.

-> imgpkg build --tag $TAG --to-tar example.tar -f $PATH_TO_FILE

[vrabbi]

any updates on this functionality being added? this would be very helpful for our development flow.

[cppforlife]

@aaronshurley is this in our prioritized backlog somewhere?

[aaronshurley]

I moved it to our prioritized backlog since this enhancement has built up a demand. We likely won't be able to start this work until at the earliest mid-to-late August due to existing priorities and folks taking vacation.

We would welcome contributions for this if anyone is up for it. Please let us know. @vrabbi @ZihanJiang96

[DennisDenuto]

@vrabbi Are you able to share your development flow? Curious to learn whether this feature would still be useful

[vrabbi]

Im very interested in this feature
For me the ask would be that the referenced images would be in the tar ball.
Basically i would create the manifests / add them to a config folder > run kbld to generate the imgpkg lock output file > run imgpkg build/push/copy -f /folder/path --to-tar /tmp/my-bundle.tar > copy tar to airgapped environment > imgpkg copy --tar /tmp/my-bundle.tar --to-repo ...

More than happy to jump on a zoom if that would be helpful to explain my use case and answer any questions

I think an imgpkg build command makes the most sense as we are building and not pushing but thats a debatable opinion

im building in an internet accessible network and then only using it in an air gapped environment

pushing to a repo just takes time and isnt useful in this case as i am purely collecting manifests and images into the bundle so that i can bring it into an air gapped network

[pivotaljohn]

FWIW, there may be an additional use-case for this feature: carvel-dev/kbld#158.

kbld has a mode where it does not push built artifacts. Having the ability to do so with imgpkg bundles fits nicely with that mode.

[cppforlife]

been brainstorming this with @joaopapereira, we have settled on

$ imgpkg push -b docker.io/dkalinin/foo:v1.0.0 -f my-bundle/
$ imgpkg copy -b docker.io/dkalinin/foo:v1.0.0 --inflate
   # registry location contains bundle + all refed images

$ imgpkg push -b docker.io/dkalinin/foo:v1.0.0 -f my-bundle/ --to-tar bundle.tar
$ imppkg copy --tar bundle.tar --inflate
   # bundle.tar contains bundle + all refed images

(--inflate naming is up for debate). above plan also addresses #37 and makes the commands consistent whether working with registry or tarballs.

[satyampsoni]

Sounds a very good improvement. I would like to understand more about it.
Anyone here who have a good understanding of this issue can jump over a meet to discuss about it?