Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a new manifest command to support multi-architecture builds #1967

Closed
wants to merge 63 commits into from
Closed

Add a new manifest command to support multi-architecture builds #1967

wants to merge 63 commits into from

Conversation

WYGIN
Copy link
Contributor

@WYGIN WYGIN commented Nov 9, 2023 •
edited by jjbustamante
Loading

Summary

Output

Before

After

Documentation

  • Should this change be documented?
    • Yes, see #___
    • No

Related

@WYGIN
WIP: added manifest_add command
17ec0e6 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_annotate command
45f39bc 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_create command
5b75d95 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_exists command
9799f01 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_inspect command
1968518 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_push command
d8f6039 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_remove command
07b9150 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_rm command
a6355e1 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added add_manifest client
f24b167 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added annotate_manifest client
15ed584 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added IndexFactory to Client
8cf016e 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added create_manifest client
0ed954f 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added delete_manifest client
dfd4bc4 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added exists_manifest client
94ba3e3 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added inspect_manifest client
4ec02e7 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added push_manifest client
f2161a9 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added remove_manifest client
2080208 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added errors
5aa4b9f 
Signed-off-by: WYGIN <[email protected]>
@github-actions github-actions bot added the type/enhancement Issue that requests a new feature or improvement. label Nov 9, 2023
@github-actions github-actions bot added this to the 0.33.0 milestone Nov 9, 2023
dependabot bot and others added 10 commits November 9, 2023 11:06
@dependabot @WYGIN
build(deps): bump github.com/buildpacks/lifecycle from 0.17.1 to 0.17.2
d10a617 
Bumps [github.com/buildpacks/lifecycle](https://github.com/buildpacks/lifecycle) from 0.17.1 to 0.17.2.
- [Release notes](https://github.com/buildpacks/lifecycle/releases)
- [Changelog](https://github.com/buildpacks/lifecycle/blob/main/RELEASE.md)
- [Commits](buildpacks/lifecycle@v0.17.1...v0.17.2)

---
updated-dependencies:
- dependency-name: github.com/buildpacks/lifecycle
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Juan Bustamante <[email protected]>
@jjbustamante @WYGIN
Updating default lifecycle version to 0.17.2
32e5d1d 
Signed-off-by: Juan Bustamante <[email protected]>
@edmorley @WYGIN
Group minor/patch version Go Dependabot updates into one PR
d5d98bc 
Go minor/patch dependencies will now be grouped, using the new
Dependabot grouping feature:
https://github.blog/changelog/2023-08-17-grouped-version-updates-by-semantic-version-level-for-dependabot/

Major updates, as well as security updates will still be opened as
separate PRs. I've not grouped GitHub Actions update PRs, since the
volume is typically much lower for those.

In addition, the schedule has been changed from daily to weekly.

This reduces project maintenance toil (no more having to manually create
combined update PRs), plus makes it less painful for contributors to
subscribe to repository notifications (currently there is a lot of noise
from Dependabot PRs being opened/auto-rebased etc).

Signed-off-by: Ed Morley <[email protected]>
@natalieparellano @WYGIN
Add buildpacksio/pack:<version>-base images to delivery
bf88725 
Signed-off-by: Natalie Arellano <[email protected]>
@natalieparellano @WYGIN
Add floating :base tag
31654b8 
Signed-off-by: Natalie Arellano <[email protected]>
@natalieparellano @WYGIN
Ensure the run image os/arch always matches:
f70fdfe 
- the builder for `pack build`
- the previous image for `pack rebase`

Signed-off-by: Natalie Arellano <[email protected]>
@natalieparellano @WYGIN
When downloading buildpacks or extensions for pack build or `pack b…
0133424 
…uilder create`,

ensure the os/arch matches the builder

Signed-off-by: Natalie Arellano <[email protected]>
@dependabot @WYGIN
build(deps): bump the go-dependencies group with 6 updates
0848fec 
Bumps the go-dependencies group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/docker/cli](https://github.com/docker/cli) | `24.0.6+incompatible` | `24.0.7+incompatible` |
| [github.com/docker/docker](https://github.com/docker/docker) | `24.0.6+incompatible` | `24.0.7+incompatible` |
| [github.com/go-git/go-git/v5](https://github.com/go-git/go-git) | `5.9.0` | `5.10.0` |
| [github.com/google/go-cmp](https://github.com/google/go-cmp) | `0.5.9` | `0.6.0` |
| [github.com/onsi/gomega](https://github.com/onsi/gomega) | `1.28.0` | `1.29.0` |
| [golang.org/x/oauth2](https://github.com/golang/oauth2) | `0.12.0` | `0.13.0` |

Updates `github.com/docker/cli` from 24.0.6+incompatible to 24.0.7+incompatible
- [Commits](docker/cli@v24.0.6...v24.0.7)

Updates `github.com/docker/docker` from 24.0.6+incompatible to 24.0.7+incompatible
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v24.0.6...v24.0.7)

Updates `github.com/go-git/go-git/v5` from 5.9.0 to 5.10.0
- [Release notes](https://github.com/go-git/go-git/releases)
- [Commits](go-git/go-git@v5.9.0...v5.10.0)

Updates `github.com/google/go-cmp` from 0.5.9 to 0.6.0
- [Release notes](https://github.com/google/go-cmp/releases)
- [Commits](google/go-cmp@v0.5.9...v0.6.0)

Updates `github.com/onsi/gomega` from 1.28.0 to 1.29.0
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](onsi/gomega@v1.28.0...v1.29.0)

Updates `golang.org/x/oauth2` from 0.12.0 to 0.13.0
- [Commits](golang/oauth2@v0.12.0...v0.13.0)

---
updated-dependencies:
- dependency-name: github.com/docker/cli
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/go-git/go-git/v5
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/google/go-cmp
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <[email protected]>
@natalieparellano @WYGIN
Fix misleading log message when publishing a buildpack package
71821d8 
When using --publish it should say "saved to registry" not "saved to docker daemon"

Signed-off-by: Natalie Arellano <[email protected]>
@WYGIN
WIP added code to generate env files in buildConfigEnv dir
4624639 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_create command
a76513d 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_exists command
128f178 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_inspect command
d238c64 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_push command
9006ad4 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_remove command
11d416d 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added manifest_rm command
749b6fe 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added add_manifest client
8b25d63 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added annotate_manifest client
6b8f798 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added IndexFactory to Client
4aa9362 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added create_manifest client
89763fd 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added delete_manifest client
a5b1faa 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added exists_manifest client
6090d11 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added inspect_manifest client
b8f3186 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added push_manifest client
b9dd8a4 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added remove_manifest client
54c3c6c 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added errors
553f927 
Signed-off-by: WYGIN <[email protected]>

 Date:      Thu Nov 9 03:33:42 2023 +0000
@WYGIN
Merge branch 'manifest' of https://github.com/WYGIN/buildpacks-pack i…
c7180d8 
…nto manifest
@WYGIN
WIP generate mock code with 'make generate'
4bbb483 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP added tests for add_manifest
8f81b6c 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP improved add_manifest client tests
f12c849 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
WIP refactor manifest_add client
4326812 
Signed-off-by: WYGIN <[email protected]>
@WYGIN
Copy link
Contributor Author

WYGIN commented Nov 10, 2023

@jjbustamante is there any possible way to build buildpacks into images directly without cli flags from pack build <image-name> [flags] so that any local imageIndex is pushed to the registry the local images to pushed into the registry without any action from users, for now i am going to notify user about the local image that need to be pushed to registry

jjbustamante
jjbustamante reviewed Nov 10, 2023
View reviewed changes
internal/commands/manifest_create.go
panic(fmt.Sprintf("error marking insecure as hidden: %v", err))
}
cmdFlags.BoolVar(&flags.publish, "publish", false, "Publish to registry")
cmdFlags.BoolVar(&flags.all, "all", false, "Add all of the list's images if the images to add are lists/index")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually leave the complexity of dealing with recursive Image Indexes because no all registries handle that correctly, it is possible according to the OCI Image spec, but in practice it is not very useful.

jjbustamante
jjbustamante reviewed Nov 10, 2023
View reviewed changes
internal/commands/manifest_create.go
}
cmdFlags.BoolVar(&flags.publish, "publish", false, "Publish to registry")
cmdFlags.BoolVar(&flags.all, "all", false, "Add all of the list's images if the images to add are lists/index")
cmdFlags.BoolVar(&flags.amend, "amend", false, "Modify an existing list/index if one with the desired name already exists")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if I want to use the same create command to update an existing image index, I think it could be another command, if we actually need it.

jjbustamante
jjbustamante reviewed Nov 10, 2023
View reviewed changes
Copy link
Member

@jjbustamante jjbustamante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments in the manifest create commands flags

@jjbustamante
Copy link
Member

@jjbustamante is there any possible way to build buildpacks into images directly without cli flags from pack build <image-name> [flags]

I think I am not understanding what you want to do here, pack buildpack package will create a Buildpack OCI image that could be saved into the daemon, registry or in a file on disk with the .cnb extension (this is actually OCI layout format).

We do want to add capabilities to pack to handle the creation of Image Indexes from the beginning, but we are trying to define the behavior with the following RFC.

In this particular work, we just want to provide end-users with some commands to create the Image Index assuming they already have OCI images created. Check this RFC

@jjbustamante jjbustamante added experimental Issue or PR refers to an experimental feature. epic/multi-arch labels Nov 20, 2023
@jjbustamante jjbustamante removed this from the 0.33.0 milestone Nov 22, 2023
@WYGIN WYGIN closed this Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jjbustamante jjbustamante jjbustamante left review comments

Assignees
No one assigned
Labels
epic/multi-arch experimental Issue or PR refers to an experimental feature. type/enhancement Issue that requests a new feature or improvement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@WYGIN @jjbustamante @edmorley @natalieparellano @jericop