From 55f9682e75944517ed794e8571dd3a6ec842659c Mon Sep 17 00:00:00 2001 From: ashpect Date: Wed, 20 Mar 2024 16:25:42 +0530 Subject: [PATCH 1/5] Proposal : Building tarred bundles without being dependent on a OCI registry Signed-off-by: ashpect --- proposals/repo-to-tar/README.md | 102 ++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100644 proposals/repo-to-tar/README.md diff --git a/proposals/repo-to-tar/README.md b/proposals/repo-to-tar/README.md new file mode 100644 index 000000000..05566763f --- /dev/null +++ b/proposals/repo-to-tar/README.md @@ -0,0 +1,102 @@ +--- +title: "Building tarred bundles without being dependent on a OCI registry" +authors: [ "Ashish Kumar " ] +status: "draft" +approvers: [ @praveenrewar @100mik @joaopapereira ] +--- + +# + +## Problem Statement + +Presently imgpkg does not support the creation of bundles directly to local disk for sharing purposes, so as to remove the need of a registry in between. This proposal aims to add this functionality to imgpkg. + +Consider this workflow, where a user wants to create a tar file of a bundle image and then push it to a registry.
+In order to obtain a tar file, one needs to first push the bundle image to the registry and then leverage the [command](https://carvel.dev/imgpkg/docs/v0.37.x/air-gapped-workflow/#option-2-with-intermediate-tarball) for copying as tar in the air gapped workflow. + +For example, In order to obtain a tar of bundle image of examples/basic-step-2, we need run the following commands in succession :
+ +`imgpkg push -b index.docker.io/user1/simple-app-bundle:v1.0.0 -f examples/basic-step-2`
+`imgpkg copy -b index.docker.io/user1/simple-app-bundle:v1.0.0 --to-tar /tmp/my-image.tar` + +This has 3 fold issues :
+1. Dependent on using a registry to create a tar file from a bundle image. +2. The tar file is created from the bundle image, which is not required if the user just wants to create a tar file having a layer as tar with configurations/metadata. +3. The bundle or image when copied as a tar file is not OCI compliant. This proposal aims to add the possibility to create a tar that is compliant with oci image layout spec . + +## Terminology / Concepts +1. Definitions of terms used in the proposal with respect to imgpkg can be found [here.](https://carvel.dev/imgpkg/docs/v0.37.x/)
+2. Resources regarding docker save and docker load can be found [here.](https://docs.docker.com/engine/reference/commandline/save/)
+3. A good article to understand OCI Artifacts can be found [here.](https://dlorenc.medium.com/oci-artifacts-explained-8f4a77945c13) +4. Read about oci-spec in detail [here.](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) + +## Proposal + +The proposal aims to remove the 3 fold issues by giving the option to create and share a tar, which is OCI compliant and can be pushed to a registry at a later point of time.
+ +#### imgpkg + +Oci Tar Creation :
+ +`--to-oci-tar` flag will be added to imgpkg push. This command will create a tar file from the bundle image. The command will be used as follows :
+`imgpkg push -b registry.example.com/xyz -f some-folder --to-oci-tar local-oci-format.tar` + +Oci Tar Copying to the Registry :
+ +Now in order to push the tar file to the registry, we can use the already existing copy command with a new flag as follows :
+`imgpkg copy --oci-tar local-oci-format.tar --to-repo registry.example.com/abc` + +### Goals and Non-goals and Future-goals + +#### Goals +1. Making OCI Complaint tar and image :
+Create and save the image in oci format locally along with tar creation. +An example of directory structure formed from complying to [image spec](https://github.com/opencontainers/image-spec/blob/main/image-layout.md) :
+ +```md +├── blobs +│ └── sha256 +│ ├── 45e16698e6e721de091b5ecbb811eae1243fb8f2ed5dbd04a9f40cd199c355f9 +│ ├── 4da097607a3cb58407cd8c157271bb5f7462a5872d516a5bc979ec678ac30f7d +│ └── 5c2dafe3c70c13990190d643c91e9f67b8129b179257674888178868474f6511 +├── index.json +└── oci-layout +``` +where one of the files in sha256 is the actual file tar and others 2 are configs and manifests.
+ +2. Add a flag `--to-oci-tar` to `imgpkg push` to be able to creata a tar while pushing a tar with imgpkg push command. + +3. Add a flag `--oci-tar` to `imgpkg copy` to be able to copy this oci kind of tar to a registry and differentiate with the `--tar` flag. + +#### Future Goals + +1. An improvement would be to have only one flag `--tar` for `imgpkg copy` making no changes to flags but making it compatible and being smart enough to understand and differentiate between both kind of tars without any need for specification.
+ +2. Add an inflate option to also contain all refs images in the oci-tar as discussed [here](https://github.com/carvel-dev/imgpkg/issues/55#issuecomment-962209740). This will be useful and true to the name of the command `imgpkg copy` as it will copy all the refs images as well and create a self-sufficient oci-tar.
+ +### Specification / Use Cases + +Use Cases : +- User wants to build a local oci compatible tar that can be pushed at a later point without being dependent on a registry. +- While using a CI framework which allows users to share files between stages, user wants to pass this output so that it can be pushed/copied in the next stage rather than having a shared registry. +- User can do the above with Package and PackageRepository bundles + +### Other Approaches Considered + +The idea behind the usage of `imgpkg push -b my.registry.io/some-name -f some-folder --to-oci-tar local-oci-format.tar` is to concentrate in a single command all the options to create a new image, that being to the registry or directly to disk. + +I had considered not do the above but modifying the `imgpkg push command` to not require `-f folder` flag and being compatible with +`imgpkg push -b my.registry.io/some-name --to-oci-tar local-oci-format.tar` to push the tar to the registry.

+ +The reason this was dropped was it is similar to imgpkg copy command used above, although initially I had considered it because the copy command can be misleading when a user just wants to push his oci-tar.

+ + +## Open Questions +1. Do we need save command to be able to save a tar file from a bundle image or should we just use push command to do the same as it forms the tar ?
+2. If save can create a `oci-tar` and `copy` can copy/push the tar to the repo, what is the need to make it compatible with `push` command ?
+3. Should we have a single flag `--tar` for `imgpkg copy` to be able to copy both kind of tars or should we have 2 flags `--tar` and `--oci-tar` ?
+ +## Answered Questions +1. The usecase for save command is when a user only need the oci compliant tar file for sharing purposes and do not have a need to push to a registry. Since push will always require a registry for pushing even for oci-tar formation.
+2. The idea behind the usage of `imgpkg push -b my.registry.io/some-name -f some-folder --to-oci-tar local-oci-format.tar` is to concentrate in a single command all the options to create a new image, that being to the registry or directly to disk. +3. A single flag would be better as it would be more intuitive and easy to use.
\ No newline at end of file From 9d57b5de15def6f91e33c870a3f68a135bcbe8f7 Mon Sep 17 00:00:00 2001 From: ashpect Date: Thu, 18 Apr 2024 22:38:12 +0530 Subject: [PATCH 2/5] updates Signed-off-by: ashpect --- proposals/repo-to-tar/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/repo-to-tar/README.md b/proposals/repo-to-tar/README.md index 05566763f..fd6bbf59f 100644 --- a/proposals/repo-to-tar/README.md +++ b/proposals/repo-to-tar/README.md @@ -64,7 +64,7 @@ An example of directory structure formed from complying to [image spec](https:// ``` where one of the files in sha256 is the actual file tar and others 2 are configs and manifests.
-2. Add a flag `--to-oci-tar` to `imgpkg push` to be able to creata a tar while pushing a tar with imgpkg push command. +2. Add a flag `--to-oci-tar` to `imgpkg push` to be able to creata a tar while pushing a tar with imgpkg push command. `imgpkg push --to-oci-tar` will only create in disk the OCI image for the bundle itself. In the future we can have the inflate flag that might help with that use case as mentioned in the future goals.
3. Add a flag `--oci-tar` to `imgpkg copy` to be able to copy this oci kind of tar to a registry and differentiate with the `--tar` flag. @@ -72,7 +72,7 @@ where one of the files in sha256 is the actual file tar and others 2 are configs 1. An improvement would be to have only one flag `--tar` for `imgpkg copy` making no changes to flags but making it compatible and being smart enough to understand and differentiate between both kind of tars without any need for specification.
-2. Add an inflate option to also contain all refs images in the oci-tar as discussed [here](https://github.com/carvel-dev/imgpkg/issues/55#issuecomment-962209740). This will be useful and true to the name of the command `imgpkg copy` as it will copy all the refs images as well and create a self-sufficient oci-tar.
+2. Add an inflate option for both copy and push which will make the oci-image contain all refs images in the oci-tar as well. It is discussed [here](https://github.com/carvel-dev/imgpkg/issues/55#issuecomment-962209740). This will be useful as it will create a self-sufficient oci-tar similar to `imgpkg copy --to-tar` but in oci format.
### Specification / Use Cases @@ -86,7 +86,7 @@ Use Cases : The idea behind the usage of `imgpkg push -b my.registry.io/some-name -f some-folder --to-oci-tar local-oci-format.tar` is to concentrate in a single command all the options to create a new image, that being to the registry or directly to disk. I had considered not do the above but modifying the `imgpkg push command` to not require `-f folder` flag and being compatible with -`imgpkg push -b my.registry.io/some-name --to-oci-tar local-oci-format.tar` to push the tar to the registry.

+`imgpkg push -b my.registry.io/some-name --to-oci-tar local-oci-format.tar` to push the tar to the registry.
The reason this was dropped was it is similar to imgpkg copy command used above, although initially I had considered it because the copy command can be misleading when a user just wants to push his oci-tar.

@@ -97,6 +97,6 @@ The reason this was dropped was it is similar to imgpkg copy command used above, 3. Should we have a single flag `--tar` for `imgpkg copy` to be able to copy both kind of tars or should we have 2 flags `--tar` and `--oci-tar` ?
## Answered Questions -1. The usecase for save command is when a user only need the oci compliant tar file for sharing purposes and do not have a need to push to a registry. Since push will always require a registry for pushing even for oci-tar formation.
+1. We already have a imgpkg copy --to-tar for that purpose.
2. The idea behind the usage of `imgpkg push -b my.registry.io/some-name -f some-folder --to-oci-tar local-oci-format.tar` is to concentrate in a single command all the options to create a new image, that being to the registry or directly to disk. -3. A single flag would be better as it would be more intuitive and easy to use.
\ No newline at end of file +3. A single flag would be better as it would be more intuitive and easy to use, which we can work on in the future after the completion of the mvp.
\ No newline at end of file From fbb58af25a4d95c7ab8fd72d59745f640492d8b8 Mon Sep 17 00:00:00 2001 From: ashpect Date: Thu, 25 Apr 2024 02:30:27 +0530 Subject: [PATCH 3/5] all open questions resolved Signed-off-by: ashpect --- proposals/repo-to-tar/README.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/proposals/repo-to-tar/README.md b/proposals/repo-to-tar/README.md index fd6bbf59f..8f6dea5f7 100644 --- a/proposals/repo-to-tar/README.md +++ b/proposals/repo-to-tar/README.md @@ -92,11 +92,5 @@ The reason this was dropped was it is similar to imgpkg copy command used above, ## Open Questions -1. Do we need save command to be able to save a tar file from a bundle image or should we just use push command to do the same as it forms the tar ?
-2. If save can create a `oci-tar` and `copy` can copy/push the tar to the repo, what is the need to make it compatible with `push` command ?
-3. Should we have a single flag `--tar` for `imgpkg copy` to be able to copy both kind of tars or should we have 2 flags `--tar` and `--oci-tar` ?
## Answered Questions -1. We already have a imgpkg copy --to-tar for that purpose.
-2. The idea behind the usage of `imgpkg push -b my.registry.io/some-name -f some-folder --to-oci-tar local-oci-format.tar` is to concentrate in a single command all the options to create a new image, that being to the registry or directly to disk. -3. A single flag would be better as it would be more intuitive and easy to use, which we can work on in the future after the completion of the mvp.
\ No newline at end of file From c154532dc099790aa093b44c320a9324ec59e9ee Mon Sep 17 00:00:00 2001 From: ashpect Date: Thu, 25 Apr 2024 02:32:02 +0530 Subject: [PATCH 4/5] switch from draft to review Signed-off-by: ashpect --- proposals/repo-to-tar/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/repo-to-tar/README.md b/proposals/repo-to-tar/README.md index 8f6dea5f7..8ed8ccff4 100644 --- a/proposals/repo-to-tar/README.md +++ b/proposals/repo-to-tar/README.md @@ -1,7 +1,7 @@ --- title: "Building tarred bundles without being dependent on a OCI registry" authors: [ "Ashish Kumar " ] -status: "draft" +status: "review" approvers: [ @praveenrewar @100mik @joaopapereira ] --- From e22e8e24e115609b9e630ac59ff305a2110369e0 Mon Sep 17 00:00:00 2001 From: ashpect Date: Tue, 7 May 2024 22:43:22 +0530 Subject: [PATCH 5/5] add review to accepted Signed-off-by: ashpect --- proposals/repo-to-tar/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/repo-to-tar/README.md b/proposals/repo-to-tar/README.md index 8ed8ccff4..2b6e06420 100644 --- a/proposals/repo-to-tar/README.md +++ b/proposals/repo-to-tar/README.md @@ -1,7 +1,7 @@ --- title: "Building tarred bundles without being dependent on a OCI registry" authors: [ "Ashish Kumar " ] -status: "review" +status: "accepted" approvers: [ @praveenrewar @100mik @joaopapereira ] ---