Add a new `manifest` command to support multi-architecture builds #1705

husni-faiz · 2023-04-05T14:42:14Z

Hi all,

Multi-architecture support for buildpacks have been requested (#1459 | #1460 ) for a long time. Recently I have been working on this as part of the Linux Foundation Mentorship program under the mentorship of Juan Bustamante (@jjbustamante), Jerico Pena (@jericop) and Aidan Delaney (@AidanDelaney)

We would appreciate your ideas and thoughts on how we want to move forward. There is a lot of validations and test to be added along the line. We are having a weekly meetings for multi-arch support every Monday @ 10:00 AM EDT/EST. You can find the zoom link in the meeting minutes document.

A new ImageIndex/ManifestList interface is being added to imgutil module.

You may find the following resources useful.

Multi-Architecutre goals
Image Index specification
ImageIndex imgutil interface
Meeting minutes
Slack Discussion
pack manifest command
Pack issue Provide a way to specify desired platform when creating packages and builders #1459
Pack issue It would be nice to support easily creating a manifest list packages and builders #1460

Fixes #1678
Fixes #1460
Fixes #1722
Fixes #1720
Fixes #1721
Fixes #1719
Fixes #1718

jjbustamante

It is looking good, let's work on the unit testing and error handling

internal/commands/manifest_add.go

internal/commands/manifest_annotate.go

internal/commands/manifest_create.go

pkg/client/create_manifest.go

Add a new manifest command to pack root commands and add a create subcommand to the manifest command. Signed-off-by: Husni Faiz <[email protected]>

Signed-off-by: Husni Faiz <[email protected]>

Use source/sinks instead of using primitives directly Signed-off-by: Husni Faiz <[email protected]>

Signed-off-by: Husni Faiz <[email protected]>

Function templates for the following manifest sub command logics - create - add - annotate - push - remove Signed-off-by: Husni Faiz <[email protected]>

Command templates for the following manifest sub commands - Add - Annotate - Push - Remove Signed-off-by: Husni Faiz <[email protected]>

Signed-off-by: Husni Faiz <[email protected]>

pkg/client/add_manifest.go

internal/commands/manifest_push_test.go

internal/commands/manifest_push.go

natalieparellano · 2024-05-01T15:10:47Z

Great work @husni-faiz @jjbustamante :) I left a few code suggestions and a few comments, but the only ones that are truly blocking for me are

Filename should be consistent with the command implemented
Let's decide on the user interface for media types (or at least explain what we mean by 'v2s2')
User interface for push options - ideally we only accept those when there is a --publish flag or we are calling push

Edit: on the last comment, I think that is actually what we are doing. I'll leave a code suggestion...

internal/commands/manifest_create.go

Signed-off-by: Juan Bustamante <[email protected]>

jjbustamante · 2024-05-02T15:18:51Z

@natalieparellano Thanks a lot for the feedback

Filename should be consistent with the command implemented

I just refactored the files

Let's decide on the user interface for media types (or at least explain what we mean by 'v2s2')

I changed it for oci or docker default value is oci

User interface for push options - ideally we only accept those when there is a --publish flag or we are calling push

Yeah, I think what is missing here is to add some validation to accept --insecure when --publish flag is used.

Signed-off-by: Juan Bustamante <[email protected]>

pkg/index/index_factory_test.go

natalieparellano · 2024-05-06T18:57:04Z

pkg/index/index_factory_test.go

+			it("finds the index in the remote registry", func() {
+				imageIndex, err = indexFactory.FindIndex(indexRepoName)
+				h.AssertNil(t, err)
+				h.AssertNotNil(t, imageIndex)


Do we want to assert anything about the index, such as its name?

We could, but we don't have any method to expose those attributes.

pkg/index/index_factory_test.go

…ndex Signed-off-by: Juan Bustamante <[email protected]>

Implement image index interface in support of https://github.com/buildpacks/rfcs/blob/main/text/0124-pack-manifest-list-commands.md and buildpacks/pack#1705 Signed-off-by: Husni Faiz <[email protected]> Signed-off-by: Juan Bustamante <[email protected]> Signed-off-by: Natalie Arellano <[email protected]> Signed-off-by: Sai Kiran <[email protected]> Signed-off-by: Sai Kiran Maggidi <[email protected]> Signed-off-by: WYGIN <[email protected]> Signed-off-by: WYGIN <[email protected]>

Signed-off-by: Juan Bustamante <[email protected]>

natalieparellano

LGTM (blocked on #2136)

Signed-off-by: Juan Bustamante <[email protected]>

github-actions bot added this to the 0.30.0 milestone Apr 5, 2023

github-actions bot added type/chore Issue that requests non-user facing changes. type/enhancement Issue that requests a new feature or improvement. labels Apr 5, 2023

husni-faiz mentioned this pull request Apr 5, 2023

Manifest List Interface buildpacks/imgutil#191

Merged

jjbustamante reviewed May 3, 2023

View reviewed changes

internal/commands/manifest_add.go Outdated Show resolved Hide resolved

internal/commands/manifest_annotate.go Outdated Show resolved Hide resolved

internal/commands/manifest_create.go Outdated Show resolved Hide resolved

pkg/client/create_manifest.go Outdated Show resolved Hide resolved

husni-faiz force-pushed the test-manifest branch from 903e3fb to afc2669 Compare May 9, 2023 15:49

natalieparellano modified the milestones: 0.30.0, 0.31.0 May 19, 2023

github-actions bot modified the milestones: 0.31.0, 0.30.0 May 23, 2023

natalieparellano modified the milestones: 0.30.0, 0.31.0 May 30, 2023

github-actions bot modified the milestones: 0.31.0, 0.30.0 Jun 1, 2023

husni-faiz added 16 commits June 5, 2023 17:56

commands: manifest: command to handle manifests

835902d

Add a new manifest command to pack root commands and add a create subcommand to the manifest command. Signed-off-by: Husni Faiz <[email protected]>

cmd: manifest create: first PoC

ffe1f93

Signed-off-by: Husni Faiz <[email protected]>

cmd: manifest: refactor code

08ec406

Use source/sinks instead of using primitives directly Signed-off-by: Husni Faiz <[email protected]>

cmd: manifest: use ggcr library

9b2a978

Signed-off-by: Husni Faiz <[email protected]>

cmd: manifest: use imgutil

3fc79cc

Signed-off-by: Husni Faiz <[email protected]>

manifest: use pack client interface

6b6f2b0

Signed-off-by: Husni Faiz <[email protected]>

manifest_create: separate input validation and manifest creation

e3908c2

Signed-off-by: Husni Faiz <[email protected]>

client: manifest sub command logic function templates

d767b1e

Function templates for the following manifest sub command logics - create - add - annotate - push - remove Signed-off-by: Husni Faiz <[email protected]>

commands: manifest sub command templates

3c0b9cc

Command templates for the following manifest sub commands - Add - Annotate - Push - Remove Signed-off-by: Husni Faiz <[email protected]>

bugfix: manifest: create: use single char for shorthand flag

e9ca7cb

Signed-off-by: Husni Faiz <[email protected]>

manifest: create: move flag validations into command logic

3a7c262

Signed-off-by: Husni Faiz <[email protected]>

manifest: create: remove path option from remote index:

40eade3

Signed-off-by: Husni Faiz <[email protected]>

create_manifest: raise error if saving the index fails

67d5644

Signed-off-by: Husni Faiz <[email protected]>

manifest_create: add default layout path to save an index

b6b0f0d

Signed-off-by: Husni Faiz <[email protected]>

manifest: flag command as experimental

43f97b5

Signed-off-by: Husni Faiz <[email protected]>

client: create an index factory

c3ac0ce

Signed-off-by: Husni Faiz <[email protected]>

natalieparellano reviewed May 1, 2024

View reviewed changes

pkg/client/add_manifest.go Outdated Show resolved Hide resolved

natalieparellano reviewed May 1, 2024

View reviewed changes

internal/commands/manifest_push_test.go Outdated Show resolved Hide resolved

natalieparellano reviewed May 1, 2024

View reviewed changes

internal/commands/manifest_push.go Show resolved Hide resolved

natalieparellano reviewed May 1, 2024

View reviewed changes

internal/commands/manifest_create.go Outdated Show resolved Hide resolved

jjbustamante added 2 commits May 2, 2024 09:20

Fixing Natalie May 1 reviews comments

5f1a28c

Signed-off-by: Juan Bustamante <[email protected]>

fixing test

c7d4c06

Signed-off-by: Juan Bustamante <[email protected]>

jjbustamante and others added 9 commits May 2, 2024 10:46

adding client manifest push commands tests

c71db73

Signed-off-by: Juan Bustamante <[email protected]>

Merge branch 'main' into test-manifest

9ae682e

updating to docker 26 the last code from main

000fb60

Signed-off-by: Juan Bustamante <[email protected]>

organizing the manifest commands unit test

ca37f34

Signed-off-by: Juan Bustamante <[email protected]>

re-org the rm manifest tests

72ae7f2

Signed-off-by: Juan Bustamante <[email protected]>

Merge branch 'main' into test-manifest

b974278

simplifying the error handling on manifest delete/remove command

12cbadf

Signed-off-by: Juan Bustamante <[email protected]>

Merge branch 'main' into test-manifest

346fe1b

adding IndexFactory test coverage

b2ad463

Signed-off-by: Juan Bustamante <[email protected]>

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

natalieparellano reviewed May 6, 2024

View reviewed changes

pkg/index/index_factory_test.go Show resolved Hide resolved

Fixing an error when format is not specified when creating an image i…

6e7f0f2

…ndex Signed-off-by: Juan Bustamante <[email protected]>

bumping imgutil version to the one with image index implementation

09a6373

Signed-off-by: Juan Bustamante <[email protected]>

natalieparellano approved these changes May 7, 2024

View reviewed changes

jjbustamante and others added 2 commits May 7, 2024 16:11

Adding acceptance test for manifest commands

922af8b

Signed-off-by: Juan Bustamante <[email protected]>

Merge branch 'main' into test-manifest

318ad0d

Signed-off-by: Juan Bustamante <[email protected]>

jjbustamante approved these changes May 8, 2024

View reviewed changes

jjbustamante merged commit 67feb16 into buildpacks:main May 8, 2024
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new `manifest` command to support multi-architecture builds #1705

Add a new `manifest` command to support multi-architecture builds #1705

husni-faiz commented Apr 5, 2023 •

edited by jjbustamante

Loading

jjbustamante left a comment

natalieparellano commented May 1, 2024 •

edited

Loading

jjbustamante commented May 2, 2024

natalieparellano May 6, 2024

jjbustamante May 6, 2024

natalieparellano left a comment

Add a new manifest command to support multi-architecture builds #1705

Add a new manifest command to support multi-architecture builds #1705

Conversation

husni-faiz commented Apr 5, 2023 • edited by jjbustamante Loading

jjbustamante left a comment

Choose a reason for hiding this comment

natalieparellano commented May 1, 2024 • edited Loading

jjbustamante commented May 2, 2024

natalieparellano May 6, 2024

Choose a reason for hiding this comment

jjbustamante May 6, 2024

Choose a reason for hiding this comment

natalieparellano left a comment

Choose a reason for hiding this comment

Add a new `manifest` command to support multi-architecture builds #1705

Add a new `manifest` command to support multi-architecture builds #1705

husni-faiz commented Apr 5, 2023 •

edited by jjbustamante

Loading

natalieparellano commented May 1, 2024 •

edited

Loading