Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(oci-layout): support in oras cp #748

Merged
merged 32 commits into from
Jan 17, 2023
Merged

feat(oci-layout): support in oras cp #748

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
83c4af1
feat: support oci image layout in `oras cp`
qweeah Jan 13, 2023
f8d4a27
add example
qweeah Jan 13, 2023
2026d9d
add example
qweeah Jan 13, 2023
071c792
add go mod
qweeah Jan 13, 2023
c244cd5
rename
qweeah Jan 13, 2023
a5fd875
doc clean
qweeah Jan 13, 2023
d4f96d3
update license header
qweeah Jan 13, 2023
b0e8e28
add interface for printer
qweeah Jan 13, 2023
3df7915
doc clean
qweeah Jan 13, 2023
ec2f556
rename
qweeah Jan 13, 2023
07dba15
Merge remote-tracking branch 'origin_src/main' into oci-copy
qweeah Jan 13, 2023
f5bb892
update examples
qweeah Jan 13, 2023
6dbd0a6
update doc
qweeah Jan 13, 2023
d71860c
doc clean
qweeah Jan 13, 2023
6d11cef
resolve comments
qweeah Jan 13, 2023
01c5674
add wrapper type for tag status printer
qweeah Jan 13, 2023
c20e948
remove long oci image layout flag
qweeah Jan 13, 2023
3962a55
doc clean
qweeah Jan 13, 2023
33f6561
resolve comments
qweeah Jan 14, 2023
fca9db1
Merge remote-tracking branch 'origin_src/main' into oci-copy
qweeah Jan 14, 2023
2a70519
fix unit test
qweeah Jan 14, 2023
6036d0d
rename target type
qweeah Jan 16, 2023
1e463d4
simplify examples
qweeah Jan 16, 2023
c83b54b
fix bug for windows path parsing and add tests
qweeah Jan 16, 2023
b4b2a6c
fix unit test
qweeah Jan 16, 2023
1585f5a
resolve comments
qweeah Jan 16, 2023
12fc8ea
doc clean
qweeah Jan 16, 2023
87771b6
doc clean
qweeah Jan 16, 2023
6af8f50
rename parsing arguments
qweeah Jan 16, 2023
56e29f9
doc clean
qweeah Jan 16, 2023
b9f5df3
resolve comments
qweeah Jan 16, 2023
440731e
fix doc
qweeah Jan 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 38 additions & 39 deletions cmd/oras/cp.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,12 @@ import (
)

type copyOptions struct {
src option.Remote
dst option.Remote
option.Common
option.Platform
recursive bool
option.BinaryTarget

recursive bool
concurrency int
srcRef string
dstRef string
extraRefs []string
}

Expand All @@ -52,54 +49,60 @@ func copyCmd() *cobra.Command {

** This command is in preview and under development. **

Example - Copy the artifact tagged with 'v1' from repository 'localhost:5000/net-monitor' to repository 'localhost:5000/net-monitor-copy'
oras cp localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1

Example - Copy the artifact tagged with 'v1' and its referrers from repository 'localhost:5000/net-monitor' to 'localhost:5000/net-monitor-copy'
oras cp -r localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1

Example - Copy the artifact tagged with 'v1' from repository 'localhost:5000/net-monitor' to 'localhost:5000/net-monitor-copy' with certain platform
oras cp --platform linux/arm/v5 localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1

Example - Copy the artifact tagged with 'v1' from repository 'localhost:5000/net-monitor' to 'localhost:5000/net-monitor-copy' with multiple tags
oras cp localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1,tag2,tag3

Example - Copy the artifact tagged with 'v1' from repository 'localhost:5000/net-monitor' to 'localhost:5000/net-monitor-copy' with multiple tags and concurrency level tuned
oras cp --concurrency 6 localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1,tag2,tag3
Example - Copy the artifacts:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FeynmanZhou I have grouped examples by features, can you help review, thanks!

oras cp localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1 # copy between repositories
oras cp localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy # copy without tagging in the destination
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think from a customers perspective --to-oci would be to an oci registry. I feel like it would be more intuitive to say --to-file or a little worse to me --to-directory

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it more, I like the --to=format and --from=format better because that would leave the possibility of other formats other than registry and directory. Having a TGZ format would be very handy for example. Having an input of stdin or stndout might be convenient as well.

Copy link
Contributor Author

@qweeah qweeah Jan 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The oci image layout is not a format but a layout, see spec

So for short flags, how about --from-layout, --to-layout for oras cp and --layout to other commands?

--layout is equivalent to --layout type=oci. To support more layouts in the future, we can add --layout type=docker (just taking docker image layout as an example, doesn't mean we need to support it)

Copy link
Contributor Author

@qweeah qweeah Jan 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you elaborate on the scenarios of using an input of stdin? To me the UX sounds looks like below:

some-cmd | oras cp - localhost/owner/name:v1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also for uploading .tgz image layout, I might take a closer look tomorrow to see if it's already supported or what we need to do to support it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having an input of stdin or stndout might be convenient as well.

Currently, oras / oras-go requires random access for read and write for performance considerations, and thus stdin and stdout are not supported. Maybe we can consider that after oras v1.0.0, trading convenience off performance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to get too side tracked by all the possible formats and destinations, but also don't want to design into a corner. It definitely might be convenient to support stdin/stdout for example, but maybe that should be done in another command. The tgz thing also might make more sense for another command since tgz input/output make sense for multiple repositories.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It definitely might be convenient to support stdin/stdout for example, but maybe that should be done in another command. The tgz thing also might make more sense for another command since tgz input/output make sense for multiple repositories.

@TerryHowe Can you help create enhancement issues with detailed scenario for those? We can continue discussion there.

Copy link
Member

@FeynmanZhou FeynmanZhou Jan 16, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think from a customers perspective --to-oci would be to an oci registry. I feel like it would be more intuitive to say --to-file or a little worse to me --to-directory.

@TerryHowe To avoid ambiguity, how about using --to-oci-layout and --from-oci-layout in oras copy respectively?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#751

oras cp --to-oci localhost:5000/net-monitor:v1 test:v1 # download into an OCI layout folder 'test'
oras cp --from-oci test:v1 localhost:5000/net-monitor:v1 # upload from an OCI layout folder 'test'

Example - Copy the artifact and its referrers:
oras cp -r localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1 # copy between repositories
oras cp -r --to-oci localhost:5000/net-monitor:v1 test:v1 # download into an OCI image layout folder 'test'
oras cp -r --from-oci test:v1 localhost:5000/net-monitor:v1 # upload from an OCI image layout folder 'test'

Example - Copy certain platform of an artifact:
oras cp --platform linux/arm/v5 localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:v1 # copy between repositories
oras cp --platform linux/arm/v5 --to-oci localhost:5000/net-monitor:v1 test:v1 # download into an OCI layout folder 'test'
oras cp --platform linux/arm/v5 --from-oci test:v1 localhost:5000/net-monitor:v1 # upload from an OCI layout folder 'test'

Example - Copy the artifact with multiple tags:
oras cp localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:tag1,tag2,tag3 # copy between repositories
oras cp localhost:5000/net-monitor:v1 localhost:5000/net-monitor-copy:tag1,tag2,tag3 # copy between repositories with concurrency level tuned
oras cp localhost:5000/net-monitor:v1 test:tag1,tag2,tag3 --to-oci # download into an OCI layout folder 'test'
oras cp test:v1 localhost:5000/net-monitor-copy:tag1,tag2,tag3 --from-oci # upload from an OCI layout folder 'test'
`,
Args: cobra.ExactArgs(2),
PreRunE: func(cmd *cobra.Command, args []string) error {
opts.From.FQDNReference = args[0]
refs := strings.Split(args[1], ",")
opts.To.FQDNReference = refs[0]
opts.extraRefs = refs[1:]
return option.Parse(&opts)
},
RunE: func(cmd *cobra.Command, args []string) error {
opts.srcRef = args[0]
refs := strings.Split(args[1], ",")
opts.dstRef = refs[0]
opts.extraRefs = refs[1:]
return runCopy(opts)
},
}

cmd.Flags().BoolVarP(&opts.recursive, "recursive", "r", false, "recursively copy the artifact and its referrer artifacts")
opts.src.ApplyFlagsWithPrefix(cmd.Flags(), "from", "source")
opts.dst.ApplyFlagsWithPrefix(cmd.Flags(), "to", "destination")
cmd.Flags().IntVarP(&opts.concurrency, "concurrency", "", 3, "concurrency level")
option.ApplyFlags(&opts, cmd.Flags())

return cmd
}

func runCopy(opts copyOptions) error {
ctx, _ := opts.SetLoggerLevel()

// Prepare source
src, err := opts.src.NewRepository(opts.srcRef, opts.Common)
src, err := opts.From.NewReadonlyTarget(ctx, opts.Common)
if err != nil {
return err
}
if opts.From.Reference == "" {
return errors.NewErrInvalidReferenceStr(opts.From.FQDNReference)
}

// Prepare destination
dst, err := opts.dst.NewRepository(opts.dstRef, opts.Common)
dst, err := opts.To.NewTarget(opts.Common)
if err != nil {
return err
}
Expand All @@ -121,14 +124,10 @@ func runCopy(opts copyOptions) error {
return display.PrintStatus(desc, "Exists ", opts.Verbose)
}

if src.Reference.Reference == "" {
return errors.NewErrInvalidReference(src.Reference)
}

var desc ocispec.Descriptor
if ref := dst.Reference.Reference; ref == "" {
if ref := opts.To.Reference; ref == "" {
// push to the destination with digest only if no tag specified
desc, err = src.Resolve(ctx, src.Reference.Reference)
desc, err = src.Resolve(ctx, opts.From.Reference)
if err != nil {
return err
}
Expand All @@ -139,27 +138,27 @@ func runCopy(opts copyOptions) error {
}
} else {
if opts.recursive {
desc, err = oras.ExtendedCopy(ctx, src, opts.srcRef, dst, opts.dstRef, extendedCopyOptions)
desc, err = oras.ExtendedCopy(ctx, src, opts.From.Reference, dst, opts.To.Reference, extendedCopyOptions)
} else {
copyOptions := oras.CopyOptions{
CopyGraphOptions: extendedCopyOptions.CopyGraphOptions,
}
if opts.Platform.Platform != nil {
copyOptions.WithTargetPlatform(opts.Platform.Platform)
}
desc, err = oras.Copy(ctx, src, opts.srcRef, dst, opts.dstRef, copyOptions)
desc, err = oras.Copy(ctx, src, opts.From.Reference, dst, opts.To.Reference, copyOptions)
}
}
if err != nil {
return err
}

fmt.Println("Copied", opts.srcRef, "=>", opts.dstRef)
fmt.Printf("Copied %s => %s \n", opts.From.FullReference(), opts.To.FullReference())
qweeah marked this conversation as resolved.
Show resolved Hide resolved

if len(opts.extraRefs) != 0 {
tagNOpts := oras.DefaultTagNOptions
tagNOpts.Concurrency = opts.concurrency
if err = oras.TagN(ctx, &display.TagManifestStatusPrinter{Repository: dst}, opts.dstRef, opts.extraRefs, tagNOpts); err != nil {
if _, err = oras.TagN(ctx, &display.TagManifestStatusPrinter{Target: dst}, opts.To.Reference, opts.extraRefs, tagNOpts); err != nil {
return err
}
}
Expand Down
34 changes: 30 additions & 4 deletions cmd/oras/internal/display/print.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ import (
"sync"

ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2"
"oras.land/oras-go/v2/content"
"oras.land/oras-go/v2/registry/remote"
"oras.land/oras-go/v2/registry"
)

var printLock sync.Mutex
Expand Down Expand Up @@ -75,13 +76,38 @@ func PrintSuccessorStatus(ctx context.Context, desc ocispec.Descriptor, status s
}

type TagManifestStatusPrinter struct {
*remote.Repository
oras.Target
registry.ReferenceFetcher
registry.ReferencePusher
}

// PushReference overrides Repository.PushReference method to print off which tag(s) were added successfully.
func (p *TagManifestStatusPrinter) PushReference(ctx context.Context, expected ocispec.Descriptor, content io.Reader, reference string) error {
if err := p.Repository.PushReference(ctx, expected, content, reference); err != nil {
return err
if repo, ok := p.Target.(registry.ReferencePusher); ok {
if err := repo.PushReference(ctx, expected, content, reference); err != nil {
return err
}
} else {
if err := p.Target.Tag(ctx, expected, reference); err != nil {
return err
}
}
qweeah marked this conversation as resolved.
Show resolved Hide resolved
return Print("Tagged", reference)
}

// FetchReference implements registry.ReferenceFetcher.
func (p *TagManifestStatusPrinter) FetchReference(ctx context.Context, reference string) (ocispec.Descriptor, io.ReadCloser, error) {
if repo, ok := p.Target.(registry.ReferenceFetcher); ok {
return repo.FetchReference(ctx, reference)
} else {
desc, err := p.Resolve(ctx, reference)
if err != nil {
return ocispec.Descriptor{}, nil, nil
}
rc, err := p.Fetch(ctx, desc)
if err != nil {
return ocispec.Descriptor{}, nil, nil
}
return desc, rc, err
}
}
5 changes: 5 additions & 0 deletions cmd/oras/internal/errors/errors.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,10 @@ import (

// NewErrInvalidReference creates a new error based on the reference string.
func NewErrInvalidReference(ref registry.Reference) error {
return NewErrInvalidReferenceStr(ref.String())
}

// NewErrInvalidReferenceStr creates a new error based on the reference string.
func NewErrInvalidReferenceStr(ref string) error {
return fmt.Errorf("%s: invalid image reference, expecting <name:tag|name@digest>", ref)
}
Loading