Add initial support for an explicit "ManifestListTags" field #2

tianon · 2017-01-24T21:29:49Z

Things we know we'll need to be able to do with the new ManifestListTags (and are thus included in this initial file format support):

get list of all manifest list tags from a manifest (manifest.GetAllManifestListTags())
get list of manifest list tags from an entry (entry.ManifestListTags)
get list of entries given a manifest list tag (manifest.GetManifestListTag(tag))

This is the first step towards docker-library/official-images#2289.

An example of the expected usage (in the hello-world image):

Maintainers: Tianon Gravi <[email protected]> (@tianon),
             Joseph Ferguson <[email protected]> (@yosifkit)
GitRepo: https://github.com/docker-library/hello-world.git

Tags: linux
ManifestListTags: latest
GitCommit: bdee60d7ff6b98037657dc34a10e9ca4ffd6785f
Directory: hello-world

Tags: nanoserver
ManifestListTags: latest
GitCommit: 1f13a5bc3b787747eeefb3b0051d8d29f851260d
Directory: hello-world/nanoserver
Constraints: nanoserver

And again as a diff, so that the actual changes are more obvious:

diff --git a/library/hello-world b/library/hello-world
index e9364fd..ef1f40c 100644
--- a/library/hello-world
+++ b/library/hello-world
@@ -4,11 +4,13 @@ Maintainers: Tianon Gravi <[email protected]> (@tianon),
              Joseph Ferguson <[email protected]> (@yosifkit)
 GitRepo: https://github.com/docker-library/hello-world.git
 
-Tags: latest
+Tags: linux
+ManifestListTags: latest
 GitCommit: bdee60d7ff6b98037657dc34a10e9ca4ffd6785f
 Directory: hello-world
 
 Tags: nanoserver
+ManifestListTags: latest
 GitCommit: 1f13a5bc3b787747eeefb3b0051d8d29f851260d
 Directory: hello-world/nanoserver
 Constraints: nanoserver

cc @friism @StefanScherer @yosifkit

(On the bashbrew side, when this change is integrated, the idea is that there will be a fourth build step after bashbrew build, bashbrew tag, and bashbrew push which is responsible for assembling and updating the actual manifest list objects on the Docker Hub so that it can be re-invoked as images are built and made available and the manifest list objects can be re-assembled as the artifacts they need to include actually exist.)

tianon · 2017-01-24T21:34:32Z

Here's a more verbose example (golang):

diff --git a/library/golang b/library/golang
index 7846547..798e030 100644
--- a/library/golang
+++ b/library/golang
@@ -5,7 +5,8 @@ Maintainers: Tianon Gravi <[email protected]> (@tianon),
              Johan Euphrosine <[email protected]> (@proppy)
 GitRepo: https://github.com/docker-library/golang.git
 
-Tags: 1.6.4, 1.6
+Tags: 1.6.4-jessie, 1.6-jessie
+ManifestListTags: 1.6.4, 1.6
 GitCommit: fddc8c18282871b554cb0d74780767690bdda3ec
 Directory: 1.6
 
@@ -27,11 +28,13 @@ Directory: 1.6/windows/windowsservercore
 Constraints: windowsservercore
 
 Tags: 1.6.4-nanoserver, 1.6-nanoserver
+ManifestListTags: 1.6.4, 1.6
 GitCommit: fddc8c18282871b554cb0d74780767690bdda3ec
 Directory: 1.6/windows/nanoserver
 Constraints: nanoserver
 
-Tags: 1.7.4, 1.7, 1, latest
+Tags: 1.7.4-jessie, 1.7-jessie, 1-jessie, jessie
+ManifestListTags: 1.7.4, 1.7, 1, latest
 GitCommit: 18ee81a2ec649dd7b3d5126b24eef86bc9c86d80
 Directory: 1.7
 
@@ -57,11 +60,13 @@ Directory: 1.7/windows/windowsservercore
 Constraints: windowsservercore
 
 Tags: 1.7.4-nanoserver, 1.7-nanoserver, 1-nanoserver, nanoserver
+ManifestListTags: 1.7.4, 1.7, 1, latest
 GitCommit: 18ee81a2ec649dd7b3d5126b24eef86bc9c86d80
 Directory: 1.7/windows/nanoserver
 Constraints: nanoserver
 
-Tags: 1.8rc2, 1.8
+Tags: 1.8rc2-jessie, 1.8-jessie
+ManifestListTags: 1.8rc2, 1.8
 GitCommit: da6e73b4fa5b3094a23202cca088dd11b74ed110
 Directory: 1.8
 
@@ -83,6 +88,7 @@ Directory: 1.8/windows/windowsservercore
 Constraints: windowsservercore
 
 Tags: 1.8rc2-nanoserver, 1.8-nanoserver
+ManifestListTags: 1.8rc2, 1.8
 GitCommit: da6e73b4fa5b3094a23202cca088dd11b74ed110
 Directory: 1.8/windows/nanoserver
 Constraints: nanoserver

tianon · 2017-01-24T22:16:17Z

cc @estesp too (maybe you've got additional thoughts you can spare on our approach? 🙏 ❤️)

estesp · 2017-01-25T04:13:03Z

Coming in without much knowledge of the official build tooling, but it seems sane from a general perspective. I can at least visualize how you use this information to do your arch-specific builds, followed by assembly into manifest list objects with the appropriate "overall" tag.

In other news, exciting to see this move forward--let me know if I can help in any way! ❤️

StefanScherer · 2017-01-25T07:39:10Z

A great step forward.
So your plan to use same version tags on Windows is to prefer the nanoserver variant? 👍

tianon · 2017-01-25T18:05:31Z

@StefanScherer yeah, probably, although looking at golang, I'm thinking that'll be something to determine case-by-case -- when I initially talked about this with some Microsoft folks, we discussed making Nano Server the default, but in the case of golang, I think windowsservercore is probably a more appropriate default (since it includes git as well, and is more "full-fledged" as a development environment), but in the case of something like, say, mongo, having Nano Server be the default definitely makes perfect sense 👍

@estesp ❤️ ❤️

tianon · 2017-01-25T22:27:21Z

From today's OCI call:

I'm even open to renaming "manifest list" to "index" -- I've never really been a fan of the "manifest list" name.

So given that manifest lists might not stay called that (potentially -- was an offhand comment, not anything concrete), I'm wondering whether perhaps we should think about choosing a more generic name for this field here that reflects the "shared" nature of it without being so tightly coupled to the name of the specific object it ends up being created as (something like SharedTags or OverlappingTags).

tianon · 2017-01-30T22:31:03Z

Off-hand comments confirmed! opencontainers/image-spec#533 (comment) (manifest lists are going to be renamed)

tianon · 2017-02-06T23:50:32Z

Ok, renamed everything to be called SharedTags instead (which is more descriptive for our purposes anyhow). 👍

tianon · 2017-02-06T23:55:16Z

There's one more thing I want to be able to do with shared tags that I can't do trivially here yet, but it's a bit complex to implement so I haven't even tried yet (but I want to make note of it here for both future reference and in case someone has an idea for an easy way to implement it).

I'd like to have some way to get unique groupings of these based on what they point to. In other words, I'd like to be able to generate Markdown similar to the following: (assuming in this contrived example that we made 1.6 a "shared tag" for the purposes of consistency, but it has no Windows variant)

Supported tags and respective `Dockerfile` links

Shared Tags

1.7.4, 1.7, 1, latest:
- 1.7.4-debian (1.7/Dockerfile)
- 1.7.4-nanoserver (1.7/windows/nanoserver/Dockerfile)
1.6.4, 1.6:
- 1.6.4-debian (1.6/Dockerfile)
debian:
- 1.7.4-debian (1.7/Dockerfile)

Simple Tags

yosifkit · 2017-02-07T00:18:07Z

manifest/rfc2822.go

@@ -249,7 +322,7 @@ func Parse2822(readerIn io.Reader) (*Manifest2822, error) {
 	if invalidMaintainers := manifest.Global.InvalidMaintainers(); len(invalidMaintainers) > 0 {
 		return nil, fmt.Errorf("invalid Maintainers: %q (expected format %q)", strings.Join(invalidMaintainers, ", "), MaintainersFormat)
 	}
-	if len(manifest.Global.Tags) > 0 {
+	if len(manifest.Global.Tags) > 0 || len(manifest.Global.SharedTags) > 0 {


Wouldn't global SharedTags be valid and useful if there is only one version of stuff? 🤔

Maintainers: Tianon Gravi <[email protected]> (@tianon), Joseph Ferguson <[email protected]> (@yosifkit) GitRepo: https://github.com/docker-library/hello-world.git SharedTags: latest Tags: linux GitCommit: bdee60d7ff6b98037657dc34a10e9ca4ffd6785f Directory: hello-world Tags: nanoserver GitCommit: 1f13a5bc3b787747eeefb3b0051d8d29f851260d Directory: hello-world/nanoserver Constraints: nanoserver

In all seriousness though, good point. I'll update.

Things we know we'll need to be able to do with the new `SharedTags` (and are thus included in this initial file format support): - get list of all shared tags from a manifest (`manifest.GetAllSharedTags()`) - get list of shared tags from an entry (`entry.SharedTags`) - get list of entries given a shared tag (`manifest.GetSharedTag(tag)`)

yosifkit · 2017-02-10T00:57:04Z

Some pseudo code to do the shared tag listing:

loop over entrys to get uniq list of shared tags
foreach sharedtag, get first tag of each entry that has it
save to map[tags_list] = [sharedTag, ...]

yosifkit · 2017-02-10T18:55:11Z

manifest/rfc2822.go

+		inter[interKey] = append(inter[interKey], sharedTag)
+	}
+	ret := []SharedTagGroup{}
+	for tags, sharedTags := range inter {


RANDOM! 💩

Doh, good point -- this does randomize the order (which won't be so great for generating deterministic documentation)...

…have some SharedTags examples

tianon · 2017-02-10T19:07:45Z

manifest/example.go

+			fmt.Printf("    - %s\n", entry.TagsString())
+		}
+	}
+	fmt.Printf("\n")


Here's the output of this, for reference:

Shared Tag Groups: - latest - 1.6.1, 1.6, 1 - 1.5.3, 1.5 - 1.5.3-debian, 1.5-debian - 1.5.3, 1.5

🤘

tianon mentioned this pull request Jan 24, 2017

Support manifest lists / manifest-tool docker-library/official-images#2289

Closed

7 tasks

tianon force-pushed the manifest-list-tags branch from ca950c5 to 06eab0b Compare January 24, 2017 21:41

yosifkit mentioned this pull request Feb 6, 2017

build for armhf nodejs/docker-node#325

Closed

tianon force-pushed the manifest-list-tags branch 2 times, most recently from df7782f to 530519c Compare February 6, 2017 23:49

yosifkit reviewed Feb 7, 2017

View reviewed changes

tianon force-pushed the manifest-list-tags branch from 530519c to 8488b2f Compare February 7, 2017 00:25

tianon assigned yosifkit Feb 8, 2017

First pass at implementing GetSharedTagGroups

c48fd9a

yosifkit reviewed Feb 10, 2017

View reviewed changes

Fix GetSharedTagGroups ordering and update our hacky example code to …

ff63f9d

…have some SharedTags examples

tianon commented Feb 10, 2017

View reviewed changes

yosifkit approved these changes Feb 10, 2017

View reviewed changes

yosifkit merged commit 5ab0344 into docker-library:master Feb 10, 2017

yosifkit deleted the manifest-list-tags branch February 10, 2017 19:14

tianon mentioned this pull request Feb 23, 2017

POWER and Z (among others) openSUSE/docker-containers-build#22

Closed

tianon mentioned this pull request Sep 13, 2017

Add SharedTags information to "Supported tags" listing docker-library/docs#1003

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial support for an explicit "ManifestListTags" field #2

Add initial support for an explicit "ManifestListTags" field #2

tianon commented Jan 24, 2017

tianon commented Jan 24, 2017

tianon commented Jan 24, 2017

estesp commented Jan 25, 2017

StefanScherer commented Jan 25, 2017

tianon commented Jan 25, 2017

tianon commented Jan 25, 2017 •

edited

Loading

tianon commented Jan 30, 2017

tianon commented Feb 6, 2017

tianon commented Feb 6, 2017 •

edited

Loading

yosifkit Feb 7, 2017

tianon Feb 7, 2017

tianon Feb 7, 2017

yosifkit commented Feb 10, 2017

yosifkit Feb 10, 2017

tianon Feb 10, 2017

tianon Feb 10, 2017

Add initial support for an explicit "ManifestListTags" field #2

Add initial support for an explicit "ManifestListTags" field #2

Conversation

tianon commented Jan 24, 2017

tianon commented Jan 24, 2017

tianon commented Jan 24, 2017

estesp commented Jan 25, 2017

StefanScherer commented Jan 25, 2017

tianon commented Jan 25, 2017

tianon commented Jan 25, 2017 • edited Loading

tianon commented Jan 30, 2017

tianon commented Feb 6, 2017

tianon commented Feb 6, 2017 • edited Loading

Supported tags and respective Dockerfile links

Shared Tags

Simple Tags

yosifkit Feb 7, 2017

Choose a reason for hiding this comment

tianon Feb 7, 2017

Choose a reason for hiding this comment

tianon Feb 7, 2017

Choose a reason for hiding this comment

yosifkit commented Feb 10, 2017

yosifkit Feb 10, 2017

Choose a reason for hiding this comment

tianon Feb 10, 2017

Choose a reason for hiding this comment

tianon Feb 10, 2017

Choose a reason for hiding this comment

tianon commented Jan 25, 2017 •

edited

Loading

tianon commented Feb 6, 2017 •

edited

Loading

Supported tags and respective `Dockerfile` links