Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pack build command to export to OCI layout format on disk #1596

Conversation

jjbustamante
Copy link
Member

@jjbustamante jjbustamante commented Jan 11, 2023

Summary

This PR expose the functionality to end users to being able to export the application image to disk in OCI layout format. The final application image can be configured to contain all the blobs from the run-image (default behavior) or using the -sparse flag the blobs from the run-image will be omitted.

Output

full image saved on disk

Let's suppose the application was built using the following command

> pack build oci:my-java-app --builder cnbs/jbustamantevmware-builder:bionic --verbose --path /Users/jbustamante/workspace/buildpack.io/samples/apps/java-maven --trust-builder

lifecycle output ...

Saving /layout-repo/index.docker.io/library/my-java-app/latest...
*** Images (/layout-repo/index.docker.io/library/my-java-app/latest@sha256:b20d9a4baac8198e2f2621a4142376a934ec9308b7d2b82dd21b4edf92629176):
      /layout-repo/index.docker.io/library/my-java-app/latest

*** Manifest Size: 1660
Reading buildpack directory: /layers/samples_java-maven
Reading buildpack directory item: jdk
Reading buildpack directory item: jdk.toml
Reading buildpack directory item: launch.toml
Reading buildpack directory item: maven_m2
Reading buildpack directory item: maven_m2.toml
Reusing tarball for layer "samples/java-maven:jdk" with SHA: sha256:a940b3489712c1f762caef8b37675f15364a65f10fb7d3f2299109f7a637afef
Adding cache layer 'samples/java-maven:jdk'
Layer 'samples/java-maven:jdk' SHA: sha256:a940b3489712c1f762caef8b37675f15364a65f10fb7d3f2299109f7a637afef
Adding cache layer 'samples/java-maven:maven_m2'
Layer 'samples/java-maven:maven_m2' SHA: sha256:8bc9932ebe39f940fc9f7ca4a608bd0598b26fc7cd2adee5728638d81d43e6c0
Successfully built image my-java-app

We can see the application was saved on disk in OCI layout format at folder my-java-app

> tree my-java-app
my-java-app
├── blobs
│   └── sha256
│       ├── 1251fdfda1211eae488b85b9d6878da34885eaae539a5db22021a2a6e0d8ddc6
│       ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8
│       ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e
│       ├── 630eb8c42eff26fe99796f8ac95d4cbf68c9840911b904c8f70cfd08bc88aee7
│       ├── 6898e29a423a34b0482cf77292814e4009aaae456e461301e4997af1e27084be
│       ├── 6c5215613ce3919ce4eebeb6497aa402cd171a1f41f6f812363c7180e2238972
│       ├── 72d9f18d70f395ff9bfae4d193077ccea3ca583e3da3dd66f5c84520c0100727
│       ├── 9aff7610b7790d95c66d6216d976a90452f92058a45343958a4b19d9d277d493
│       ├── a99cbc83e1aaf58380e98988983afbf463b055852462b24c8b609100b74bde75
│       ├── b20d9a4baac8198e2f2621a4142376a934ec9308b7d2b82dd21b4edf92629176
│       └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78
├── index.json
└── oci-layout

2 directories, 13 files

image saved on disk avoiding run-image blobs

On the other hand, if we used the --sparse flag to export the image

> pack build oci:my-java-app --builder cnbs/jbustamantevmware-builder:bionic --verbose --path /Users/jbustamante/workspace/buildpack.io/samples/apps/java-maven --trust-builder --sparse

The output do not contain the run-image blobs

> tree my-java-app
my-java-app
├── blobs
│   └── sha256
│       ├── 2fa192256ce255c6ea6c1296eadfe2feba8094f40e6aa85e699645caca2e85d8
│       ├── 5a44e4f7b58d74fe6f92dd7028075c91191128d1e2e7f39846fe061a9a98836e
│       ├── 6622426666a7b61c086c84203082d5f64495be1f8b085137e53b0554cfcdb50a
│       ├── 6c5215613ce3919ce4eebeb6497aa402cd171a1f41f6f812363c7180e2238972
│       ├── a99cbc83e1aaf58380e98988983afbf463b055852462b24c8b609100b74bde75
│       ├── d51dbef8a63d4ca36ef0a358e2e7e77cfffc2687644f5cf9d1844d3b8b11c412
│       ├── f53c5083455f65f6e262b356e15a889e0f351f6c725aa3031103c27e3b15190a
│       └── f9d6350d0c44c0e7165a522155f53181ce8c163a6b8ead1f6baea22d1a8d8a78
├── index.json
└── oci-layout

2 directories, 10 files

Before

The feature doesn't exist in previous version of pack

After

With this new PR, pack will have the following new behavior:

  • OCI layout feature is experimental, in order to be used, it must be enabled by pack users running pack config experimental true
  • Once the experimental mode is enabled, the user can enable the feature using oci:<image-path> when pack build is invoked
  • If the user wants to prevent run-image blobs to be written on disk, they can passthrough the flag --sparse
  • For now, pack will created a folder at /$HOME/.pack/layout-repo to save the run-image in OCI layout, this path will be mounted during lifecycle build execution.

Documentation

  • Should this change be documented?

Related

Resolves #1548

@github-actions github-actions bot added type/chore Issue that requests non-user facing changes. type/enhancement Issue that requests a new feature or improvement. labels Jan 11, 2023
@github-actions github-actions bot added this to the 0.29.0 milestone Jan 11, 2023
@jjbustamante jjbustamante force-pushed the enhancement/issue-1548-imagee-in-oci-layout-format branch 2 times, most recently from 52af388 to 86618f2 Compare January 11, 2023 19:17
@github-actions github-actions bot removed the type/chore Issue that requests non-user facing changes. label Jan 11, 2023
@jjbustamante jjbustamante force-pushed the enhancement/issue-1548-imagee-in-oci-layout-format branch 2 times, most recently from c1c247a to 1f61446 Compare January 17, 2023 00:56
@github-actions github-actions bot added the type/chore Issue that requests non-user facing changes. label Jan 17, 2023
@codecov
Copy link

codecov bot commented Jan 17, 2023

Codecov Report

Merging #1596 (005aed5) into main (a803a52) will decrease coverage by 0.47%.
The diff coverage is 26.93%.

❗ Current head 005aed5 differs from pull request most recent head 256b0dc. Consider uploading reports for the commit 256b0dc to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1596      +/-   ##
==========================================
- Coverage   79.72%   79.25%   -0.47%     
==========================================
  Files         165      147      -18     
  Lines       10885     9076    -1809     
==========================================
- Hits         8677     7192    -1485     
+ Misses       1683     1463     -220     
+ Partials      525      421     -104     
Flag Coverage Δ
os_linux 79.09% <26.93%> (-0.41%) ⬇️
os_macos 76.83% <26.93%> (-0.79%) ⬇️
os_windows 79.15% <26.93%> (-0.46%) ⬇️
unit 79.25% <26.93%> (-0.47%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@jjbustamante jjbustamante force-pushed the enhancement/issue-1548-imagee-in-oci-layout-format branch 2 times, most recently from 31b8178 to f0666c1 Compare February 6, 2023 21:10
Signed-off-by: Juan Bustamante <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
…, app image destination and previous image) is mounting independently

Signed-off-by: Juan Bustamante <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
@jjbustamante jjbustamante force-pushed the enhancement/issue-1548-imagee-in-oci-layout-format branch from 3fa4b03 to 256b0dc Compare February 17, 2023 17:09
@jjbustamante jjbustamante marked this pull request as ready for review February 23, 2023 17:48
@jjbustamante jjbustamante requested review from a team as code owners February 23, 2023 17:48
Comment on lines +37 to +41
if cfg.Experimental {
cfg.LayoutRepositoryDir = filepath.Join(filepath.Dir(cfgPath), "layout-repo")
} else {
cfg.LayoutRepositoryDir = ""
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels a little odd to me. Are we sure we want the default layout dir in the config dir?

Also, it looks like this isn't actually configurable right? It's either the default or nothing.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, for now, because the feature is experimental, I didn't want to do a lot of work giving the user a configuration option. basically, the idea is:

  • We need a place to save the run-image on disk, maybe the user doesn't care, that's why I am saving them in the pack home directory
  • I am planning to document this behavior in the docs so users can be aware of it.

The other alternative is, I could work on the user experience for configuring and customizing this kind of thing, but, I was planing to do it after shipping at least something the users can play with

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I can see pack already save some data in the same folder, for example.

➜  > tree ~/.pack -L 1
/Users/jbustamante/.pack
├── completion.zsh
├── config.toml
├── download-cache
├── layout-repo
├── registry-a932275bd19c2d9e1b88fa06698fd2f5427a363d25bf87fa500691c373089381
├── registry2908354347
└── registry57772754

6 directories, 2 files

Those registry-* folders are already saving data there

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it just metadata that goes in this dir? Or does it actually store large files?

Copy link
Member Author

@jjbustamante jjbustamante Feb 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will save the run-image in OCI layout format, it could have all the blobs depending on whether the user set the sparse flag or not. But it could have large files, for sure

@jkutner jkutner enabled auto-merge March 7, 2023 15:12
@jjbustamante jjbustamante force-pushed the enhancement/issue-1548-imagee-in-oci-layout-format branch from 0ac038e to 12c370a Compare March 8, 2023 17:41
Signed-off-by: Juan Bustamante <[email protected]>
@jkutner jkutner merged commit 0f0918d into buildpacks:main Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/chore Issue that requests non-user facing changes. type/enhancement Issue that requests a new feature or improvement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pack build to export to OCI layout format on disk
2 participants