Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(e2e): add tests to cover OCI spec v1.1.0-rc.4 and OCI distribution spec 1.1.0-rc3 #1099

Merged
merged 73 commits into from
Sep 1, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
f61b791
test(e2e): add zot
qweeah Aug 22, 2023
06abf60
add auth tests
qweeah Aug 22, 2023
8e644c6
uncomment changes
qweeah Aug 23, 2023
0c7a3ee
move auth and scenario suite to zot
qweeah Aug 23, 2023
35d8bd4
add password
qweeah Aug 23, 2023
1700e23
update description
qweeah Aug 23, 2023
7267ca4
separate prepare script out
qweeah Aug 23, 2023
a2cc4ea
check in zot config
qweeah Aug 23, 2023
e5f7398
add run access
qweeah Aug 23, 2023
2feed93
fix prepare bug
qweeah Aug 23, 2023
96fc889
remove prepare changes
qweeah Aug 24, 2023
2c2b597
run zot container via current user
qweeah Aug 25, 2023
6f9f3c2
rename zot
qweeah Aug 25, 2023
0a00292
add images repo for command suite and attach tests
qweeah Aug 25, 2023
3d60a75
update blob tests
qweeah Aug 25, 2023
c79ba60
Merge remote-tracking branch 'origin_src/main' into zot-attach
qweeah Aug 25, 2023
6e1998b
Merge remote-tracking branch 'origin_src/main' into zot-blob
qweeah Aug 28, 2023
d3d747b
Merge remote-tracking branch 'origin_src/main' into zot-attach
qweeah Aug 28, 2023
8efd185
add doc
qweeah Aug 28, 2023
f3cf89e
doc clean
qweeah Aug 28, 2023
8295d85
Merge branch 'zot-attach' into zot-blob
qweeah Aug 28, 2023
dea043d
fix e2e
qweeah Aug 28, 2023
1945b87
test(e2e): migrate `oras push` specs to ZOT
qweeah Aug 28, 2023
f028e61
fix e2e
qweeah Aug 28, 2023
a25d23f
fix e2e
qweeah Aug 29, 2023
003072a
Merge branch 'zot-push' into zot-pull
qweeah Aug 29, 2023
df78bef
test(e2e): migrate specs for `oras pull`
qweeah Aug 29, 2023
1e9bbd8
fix e2e
qweeah Aug 29, 2023
e2a4c38
Merge branch 'zot-push' into zot-pull
qweeah Aug 29, 2023
003c2c9
test(e2e): migrate `oras tag` specs to ZOT
qweeah Aug 29, 2023
e654e06
test(e2e): migrate `oras tag` specs to ZOT
qweeah Aug 29, 2023
d57f9bc
test(e2e): migrate `oras repo` specs to ZOT
qweeah Aug 29, 2023
0c7187c
remove focus
qweeah Aug 29, 2023
b42c474
Merge remote-tracking branch 'origin_src/main' into zot-blob
qweeah Aug 29, 2023
1f18415
test(e2e): migrate `oras repo` specs to ZOT
qweeah Aug 29, 2023
670ac47
Merge branch 'zot-blob' into zot-repo
qweeah Aug 29, 2023
cd370e3
test(e2e): migrate `oras manifest` specs to ZOT
qweeah Aug 29, 2023
313073d
move invalid pushing to fallback distribution
qweeah Aug 29, 2023
98be084
remove focus
qweeah Aug 29, 2023
69ef80d
Merge remote-tracking branch 'origin_src/main' into zot-manifest
qweeah Aug 30, 2023
d657c2f
test(e2e): migrate `oras cp` specs to ZOT
qweeah Aug 30, 2023
c6a290d
rebake test data for distribution
qweeah Aug 30, 2023
67d0556
add foobar referrers to ZOT
qweeah Aug 30, 2023
b6e10c5
update testdata constants
qweeah Aug 30, 2023
ba8de9a
update doc
qweeah Aug 31, 2023
095da7e
Merge branch 'zot-testdata' into zot-cp
qweeah Aug 31, 2023
a75205c
fix e2e
qweeah Aug 31, 2023
eded767
add test data
qweeah Aug 31, 2023
8d8745e
add linux amd64 referrers data
qweeah Aug 31, 2023
67d31de
test(e2e): migrate `oras discover` specs to ZOT
qweeah Aug 31, 2023
df7ec84
Merge branch 'zot-discover' into zot-cp
qweeah Aug 31, 2023
88e64fb
fix e2e
qweeah Aug 31, 2023
2150f68
Merge branch 'zot-discover' into zot-cp
qweeah Aug 31, 2023
0a981d8
add cp spec back
qweeah Aug 31, 2023
a49f6e9
add old artifact consumption test
qweeah Aug 31, 2023
b87864b
trigger building
qweeah Aug 31, 2023
bc06d38
add tests to cover rc2 artifacts
qweeah Aug 31, 2023
d75b7fd
Merge branch '1.1.0-e2e' into e2e-test
qweeah Aug 31, 2023
4b84e54
rebake test data
qweeah Aug 31, 2023
a90495d
rebake test data
qweeah Aug 31, 2023
ac755e5
Merge branch 'zot-testdata' into zot-cp
qweeah Sep 1, 2023
1eb658f
revert
qweeah Sep 1, 2023
9df1e0c
Merge branch 'zot-cp' into e2e-test
qweeah Sep 1, 2023
85e9955
Merge remote-tracking branch 'origin_src/main' into e2e-test
qweeah Sep 1, 2023
0dd5b3c
Merge remote-tracking branch 'origin_src/main' into zot-cp
qweeah Sep 1, 2023
8c9958d
Merge branch 'zot-cp' into e2e-1.1.0-rc2
qweeah Sep 1, 2023
3282b7b
add example artifacts
qweeah Sep 1, 2023
296269c
add 1.0 pull test
qweeah Sep 1, 2023
413a69b
add cp and pull tests
qweeah Sep 1, 2023
064f161
add push test
qweeah Sep 1, 2023
77a3bba
Merge remote-tracking branch 'origin_src' into e2e-1.1.0-rc.4
qweeah Sep 1, 2023
7f5158e
add index with subject
qweeah Sep 1, 2023
0bb8f66
add cp test
qweeah Sep 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions test/e2e/internal/testdata/artifact/blob/const.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package blob

import "oras.land/oras/test/e2e/internal/utils/match"

const (
Tag = "blob"
)

var (
StateKeys = []match.StateKey{
{Digest: "44136fa355b3", Name: "application/vnd.oci.empty.v1+json"},
{Digest: "2ef548696ac7", Name: "hello.tar"},
{Digest: "ccd5f15fb7e0", Name: "application/vnd.oci.image.manifest.v1+json"},
}
)
30 changes: 30 additions & 0 deletions test/e2e/internal/testdata/artifact/config/const.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package config

import "oras.land/oras/test/e2e/internal/utils/match"

const (
Tag = "config"
)

var (
StateKeys = []match.StateKey{
{Digest: "fe9dbc99451d", Name: "application/vnd.oci.image.config.v1+json"},
{Digest: "2ef548696ac7", Name: "hello.tar"},
{Digest: "0e7404d1cc4a", Name: "application/vnd.oci.image.manifest.v1+json"},
}
)
36 changes: 36 additions & 0 deletions test/e2e/internal/testdata/artifact/const.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package artifact

import (
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras/test/e2e/internal/utils/match"
)

const (
DefaultArtifactType = "application/vnd.unknown.artifact.v1"
DefaultConfigMediaType = "application/vnd.oci.empty.v1+json"
)

var (
DefaultConfigStateKey = match.StateKey{Digest: "44136fa355b3", Name: "application/vnd.oci.empty.v1+json"}
EmptyLayerJSON = ocispec.Descriptor{
MediaType: "application/vnd.oci.empty.v1+json",
Digest: "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
Size: 2,
Data: []byte(`{}`),
}
)
26 changes: 26 additions & 0 deletions test/e2e/internal/testdata/artifact/index/const.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
Copyright The ORAS Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package index

import "oras.land/oras/test/e2e/internal/utils/match"

var (
ManifestDigest = "sha256:7679bc22c33b87aa345c6950a993db98a6df7a6cc77a35c388908a3a50be6bad"
ManifestStatusKey = match.StateKey{
Digest: "7679bc22c33b",
Name: "application/vnd.oci.image.index.v1+json",
}
)
1 change: 1 addition & 0 deletions test/e2e/internal/testdata/multi_arch/const.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ var (
{Digest: "2ef548696ac7", Name: "hello.tar"},
{Digest: "fe9dbc99451d", Name: "application/vnd.oci.image.config.v1+json"},
{Digest: "9d84a5716c66", Name: "application/vnd.oci.image.manifest.v1+json"},
{Digest: "e2bfc9cc6a84", Name: "application/vnd.oci.image.index.v1+json"},
}
// below are newly baked artifacts for ZOT
IndexZOTReferrerStateKey = match.StateKey{Digest: "d37baf66300b", Name: "application/vnd.oci.image.manifest.v1+json"}
Expand Down
35 changes: 29 additions & 6 deletions test/e2e/suite/command/cp.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ import (
"github.com/onsi/gomega/gbytes"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"oras.land/oras-go/v2"
"oras.land/oras/test/e2e/internal/testdata/artifact/blob"
"oras.land/oras/test/e2e/internal/testdata/artifact/config"
"oras.land/oras/test/e2e/internal/testdata/artifact/index"
"oras.land/oras/test/e2e/internal/testdata/feature"
"oras.land/oras/test/e2e/internal/testdata/foobar"
ma "oras.land/oras/test/e2e/internal/testdata/multi_arch"
Expand Down Expand Up @@ -70,9 +73,29 @@ func CompareRef(src, dst string) {

var _ = Describe("1.1 registry users:", func() {
When("running `cp`", func() {
It("should copy an artifact with blob", func() {
src := RegistryRef(ZOTHost, ArtifactRepo, blob.Tag)
dst := RegistryRef(ZOTHost, cpTestRepo("artifact-with-blob"), "copied")
ORAS("cp", src, dst, "-v").MatchStatus(blob.StateKeys, true, len(blob.StateKeys)).Exec()
CompareRef(src, dst)
})

It("should copy an artifact with config", func() {
src := RegistryRef(ZOTHost, ArtifactRepo, config.Tag)
dst := RegistryRef(ZOTHost, cpTestRepo("artifact-with-config"), "copied")
ORAS("cp", src, dst, "-v").MatchStatus(config.StateKeys, true, len(config.StateKeys)).Exec()
})

It("should copy index and its subject", func() {
stateKeys := append(ma.IndexStateKeys, index.ManifestStatusKey)
src := RegistryRef(ZOTHost, ArtifactRepo, index.ManifestDigest)
dst := RegistryRef(ZOTHost, cpTestRepo("index-with-subject"), "")
ORAS("cp", src, dst, "-v").MatchStatus(stateKeys, true, len(stateKeys)).Exec()
})

It("should copy an image to a new repository via tag", func() {
src := RegistryRef(ZOTHost, ImageRepo, foobar.Tag)
dst := RegistryRef(ZOTHost, cpTestRepo("tag"), "copiedTag")
dst := RegistryRef(ZOTHost, cpTestRepo("tag"), "copied")
ORAS("cp", src, dst, "-v").MatchStatus(foobarStates, true, len(foobarStates)).Exec()
CompareRef(src, dst)
})
Expand Down Expand Up @@ -111,7 +134,7 @@ var _ = Describe("1.1 registry users:", func() {
// validate
CompareRef(RegistryRef(ZOTHost, ImageRepo, ma.Digest), dst)
var index ocispec.Index
bytes := ORAS("discover", dst, "-o", "json").
bytes := ORAS("discover", dst, "-o", "json", "--artifact-type", ma.IndexReferrerConfigStateKey.Name).
MatchKeyWords(ma.IndexReferrerDigest).
WithDescription("copy image referrer").
Exec().Out.Contents()
Expand All @@ -136,7 +159,7 @@ var _ = Describe("1.1 registry users:", func() {
// validate
CompareRef(RegistryRef(ZOTHost, ImageRepo, ma.Digest), dst)
var index ocispec.Index
bytes := ORAS("discover", dst, "-o", "json").
bytes := ORAS("discover", dst, "-o", "json", "--artifact-type", ma.IndexReferrerConfigStateKey.Name).
MatchKeyWords(ma.IndexReferrerDigest).
WithDescription("copy image referrer").
Exec().Out.Contents()
Expand Down Expand Up @@ -454,8 +477,8 @@ var _ = Describe("OCI layout users:", func() {
dstManifest := ORAS("manifest", "fetch", dst, Flags.Layout).WithDescription("fetch from destination to validate").Exec().Out.Contents()
Expect(srcManifest).To(Equal(dstManifest))
var index ocispec.Index
bytes := ORAS("discover", dst, "-o", "json", Flags.Layout).
MatchKeyWords(ma.IndexReferrerDigest).
bytes := ORAS("discover", dst, "-o", "json", Flags.Layout, "--artifact-type", ma.IndexReferrerConfigStateKey.Name).
// MatchKeyWords(ma.IndexReferrerDigest).
WithDescription("copy image referrer").
Exec().Out.Contents()
Expect(json.Unmarshal(bytes, &index)).ShouldNot(HaveOccurred())
Expand Down Expand Up @@ -484,7 +507,7 @@ var _ = Describe("OCI layout users:", func() {
dstManifest := ORAS("manifest", "fetch", dst).WithDescription("fetch from destination to validate").Exec().Out.Contents()
Expect(srcManifest).To(Equal(dstManifest))
var index ocispec.Index
bytes := ORAS("discover", dst, "-o", "json").
bytes := ORAS("discover", dst, "-o", "json", "--artifact-type", ma.IndexReferrerConfigStateKey.Name).
MatchKeyWords(ma.IndexReferrerDigest).
WithDescription("copy image referrer").
Exec().Out.Contents()
Expand Down
67 changes: 66 additions & 1 deletion test/e2e/suite/command/pull.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ import (
. "github.com/onsi/gomega"
"github.com/onsi/gomega/gbytes"
"oras.land/oras-go/v2"
"oras.land/oras/test/e2e/internal/testdata/artifact/blob"
"oras.land/oras/test/e2e/internal/testdata/artifact/config"
"oras.land/oras/test/e2e/internal/testdata/feature"
"oras.land/oras/test/e2e/internal/testdata/foobar"
"oras.land/oras/test/e2e/internal/testdata/multi_arch"
Expand All @@ -41,7 +43,7 @@ var _ = Describe("ORAS beginners:", func() {
})
})

var _ = Describe("Remote registry users:", func() {
var _ = Describe("OCI spec 1.1 registry users:", func() {
When("pulling images from remote registry", func() {
var (
configName = "test.config"
Expand Down Expand Up @@ -105,6 +107,69 @@ var _ = Describe("Remote registry users:", func() {
ORAS("pull", RegistryRef(ZOTHost, ImageRepo, "multi"), "--platform", "linux/amd64", "-v", "-o", GinkgoT().TempDir()).
MatchStatus(multi_arch.LinuxAMD64StateKeys, true, len(multi_arch.LinuxAMD64StateKeys)).Exec()
})

It("should pull an artifact with blob", func() {
pullRoot := GinkgoT().TempDir()
ORAS("pull", RegistryRef(ZOTHost, ArtifactRepo, blob.Tag), "-v", "-o", pullRoot).Exec()
Expect(filepath.Join(pullRoot, multi_arch.LayerName)).Should(BeAnExistingFile())
})

It("should pull an artifact with config", func() {
pullRoot := GinkgoT().TempDir()
ORAS("pull", RegistryRef(ZOTHost, ArtifactRepo, config.Tag), "-v", "-o", pullRoot).Exec()
Expect(filepath.Join(pullRoot, multi_arch.LayerName)).Should(BeAnExistingFile())
})

It("should copy an artifact with blob", func() {
repo := cpTestRepo("artifact-with-blob")
stateKeys := append(append(foobarStates, foobar.ImageReferrersStateKeys...), foobar.ImageReferrerConfigStateKeys...)
src := RegistryRef(ZOTHost, ArtifactRepo, foobar.SignatureImageReferrer.Digest.String())
dst := RegistryRef(FallbackHost, repo, "")
ORAS("cp", "-r", src, dst, "-v").MatchStatus(stateKeys, true, len(stateKeys)).Exec()
CompareRef(src, RegistryRef(FallbackHost, repo, foobar.SignatureImageReferrer.Digest.String()))
ORAS("discover", "-o", "tree", RegistryRef(FallbackHost, repo, foobar.Digest)).
WithDescription("discover referrer via subject").MatchKeyWords(foobar.SignatureImageReferrer.Digest.String(), foobar.SBOMImageReferrer.Digest.String()).Exec()
})
})
})

var _ = Describe("OCI spec 1.0 registry users:", func() {
It("should pull all files in an image to a target folder", func() {
pullRoot := "pulled"
configName := "test.config"
tempDir := PrepareTempFiles()
stateKeys := append(foobar.ImageLayerStateKeys, foobar.ManifestStateKey, foobar.ImageConfigStateKey(configName))
ORAS("pull", RegistryRef(FallbackHost, ArtifactRepo, foobar.Tag), "-v", "--config", configName, "-o", pullRoot).
MatchStatus(stateKeys, true, len(stateKeys)).
WithWorkDir(tempDir).Exec()
// check config
configPath := filepath.Join(tempDir, pullRoot, configName)
Expect(configPath).Should(BeAnExistingFile())
f, err := os.Open(configPath)
Expect(err).ShouldNot(HaveOccurred())
defer f.Close()
Eventually(gbytes.BufferReader(f)).Should(gbytes.Say("{}"))
for _, f := range foobar.ImageLayerNames {
// check layers
Binary("diff", filepath.Join(tempDir, "foobar", f), filepath.Join(pullRoot, f)).
WithWorkDir(tempDir).Exec()
}

ORAS("pull", RegistryRef(FallbackHost, ArtifactRepo, foobar.Tag), "-v", "-o", pullRoot, "--keep-old-files").
ExpectFailure().
WithDescription("fail if overwrite old files are disabled")
})

It("should pull subject", func() {
tempDir := GinkgoT().TempDir()
stateKeys := append(append(
foobar.ImageLayerStateKeys,
foobar.ManifestStateKey),
foobar.ImageReferrersStateKeys...,
)
ORAS("pull", RegistryRef(FallbackHost, ArtifactRepo, foobar.SignatureImageReferrer.Digest.String()), "-v", "--include-subject").
MatchStatus(stateKeys, true, len(stateKeys)).
WithWorkDir(tempDir).Exec()
})
})

Expand Down
Loading
Loading