doc/spec: namespaces and discovery

[dmcgowan]

Includes description of namespace and discovery process.

• Add aliases
• Finalize HTTP Discovery specification
• Rename Namespace to Remotes (or better name?)
• Add UX design (modeled after git remotes?) @stevvooe
• Add examples to illustrate terminology

[titanous]

"certificate" is not specific. Do you mean an X.509 v3 certificate?

[dmcgowan]

x509 is the only thing being discussed right now. I am intentionally not specifying right here because the trust proposal is still being drafted with the specifics. I will likely track those changes then point there later.

[vbatts]

why ~ instead of `?}

[mattmoor]

Per moby/moby#13089

It would be fantastic if the grammar could be partitioned in such a fashion that discovery also captured naming restrictions for a registry. The way the v2 naming restrictions stand today, we (gcr.io) have a large number of customer scenarios that will be broken by its lack of backward compatibility.

When talking about naming, we consider backwards-compatibility a minimum bar, but would love to see some flexibility opened up here to enable registry providers some leeway. Federating the restrictions, possibly by using a discovery service like this, it one way of achieving that.

[dmcgowan]

@mattmoor when you mention lack of backward compatibility are you referring to the lack of support for <path-part> beginning with a _? The grammar is intended to be a superset while trying to avoid "bad" names such images with path parts of consecutive ._-.

[mattmoor]

I outlined a few discrepancies in the v2 grammar.

The biggest known issue is the "_" prefix, but it is very possible that the new length restrictions will affect our customers as well.

[dmcgowan]

This proposal is attempting to define a clear and wide standard for images names. We intended it to be a superset but the v1 image name specification is not clearly defined, making it hard to make a clean specification that captures all the edge cases with the names. Prefixing images with "_" is something we put in the category of a dirty edge cases (along with consecutive non-alphanumeric characters).

The intent of this proposal is not to suggest that different registries can/should support a different grammar for names, but rather the grammar is valid in any registry and image names can be uniform. If this is a problem I encourage you to join the distribution IRC channel or working group so we can discuss further the use case.

[jonboulle]

namespace

[jonboulle]

eww, smart quotes!

[dmcgowan]

very keen eye, do not know where those came from

[yosifkit]

Since the hostname is supposed to allow all valid domain names, it should not exclude valid domain names like 509films.com. See wikipedia for a simple explanation. To keep this spec and the eventual implementation simple, we can be more lax on the limit of 127 tree subdivisions and only 253 characters, letting the dns libraries handle the error.

[stevvooe]

While this is partly a hostname, first and foremost, it is a namespace for images. This has to be both a hostname and a valid name component for a docker image. Opening it up to allow whatever dns allows would be problematic for backwards compatibility.

If you disagree, please purpose a change to the BNF.

[yosifkit]

Ok, I would allow starting with a number, but requiring a letter somewhere after that. We should also allow multiple hyphens especially for punycode.

- <host-part> ::= <regexp "[a-z]([-]?[a-z0-9])*">
+ <host-part> ::= <regexp "([0-9][-0-9]*)?[a-z](-*[a-z0-9])*">
# Number, followed by optional non-letters, followed by required letter, followed by anything, then a non-hyphen character to end

valid:
9-a
xn--n3h
lib

invalid:
9
99-99
a0- (still invalid)

[yosifkit]

@stevvooe (or any others) what are the thoughts about my proposed BNF to bring the host parts in line with DNS names? (ie the comment above this one)

I have not seen any discussion on this and wanted to make sure we are not limiting where a user can host their own registry by only allowing a subset of valid domain names.

It may require tweaking if you want to follow the exact DNS specifications, but I would rather we at least err on being a superset of DNS rather than a subset.

[stevvooe]

@yosifkit A few constraints on what is accepted are acceptable in the face of consistency and security. I am not sure that I agree that this would limit where a user can host a registry. Domains are cheap or free. Do you have specific examples of situations where people want to host images in a domain that can't be represented and they absolutely cannot change the domain?

Check out #609 for the work on the reference package. We are putting together a full BNF for a "Reference" which will capture all of this.

Either way, it's easier to make this more accepting later than it is to lock this down further in the future. We can make this suggested change at any time, but going back to the more restrictive format would be impossible.

[stevvooe]

@paultag That's a bit oversimplified. There are a number of factors.

@yosifkit @tianon Right now, you are literally bike-shedding over "I want a snowman". Reducing my argument to 'arbitrarily restricting people's options... because "Domains are cheap or free"' is not really fair.

This specification captures the current state of "namespaces" and extends it with discovery. This isn't the right venue for changes to this format. If you would like to make a change here, the following are list of changes we've made to allowed image names based on clear, well-founded technical arguments:

moby/moby#10392
#687
#241

I look forward to your constructive, well-founded PR to add punycode support to image names! We'll update this proposal accordingly.

[dmcgowan]

Been trying to stay out of this one. I think @stevvooe is right that this is not the place to make such changes and decisions. This format is tracking a single DNS RFC. If we want to include other standard formats we should make sure the support is elsewhere and document what RFCs we are using to build this BNF. Then it can easily be audited for accuracy. For the <path-part> we do have more domain over other changes since there are no RFCs to base this off of.

https://www.youtube.com/watch?v=C1ZHetiuRb0

[tianon]

Ok, fair enough; thanks. 👍

[yosifkit]

Using docker 1.7.1 and registry:2.0.1 I was able to tag, push, and pull an image as 9n--13h.docker:5000/hello-world*. So it is not an active problem in docker or the registry, just, as I understand it, a proposed definition here. The reason I initially chimed in was to ensure that this definition did not limit what is an acceptable hostname. So, yes, I want the BNF to reflect the DNS RFCs to allow all valid DNS hostnames as you state in this doc:

The host name follows the DNS rules without change. link

Also, it looks like my proposed change above is wrong in some places. A better one would be ^(?![0-9]+$)(?!-)[a-zA-Z0-9-]{1,63}(?<!-)$ (stackoverflow).

_{* yes, that is not a punycode address, but highlights the needed double hyphen for punycode as well as domains that start with a number that also contain a non-number}

[stevvooe]

@yosifkit @tianon I've added a unit test for a few of these examples that we support in the future to ensure any changes get caught in the implementation of this specification.

#791

Feel free to submit a PR that modifies the name component regexp appriopriately and we can flip this test case to valid.

[stevvooe]

We are planning on reexamining our approach here. Closing this PR for now.

[jonboulle]

@stevvooe is there an alternative proposal that interested parties can follow along?

[stevvooe]

@jonboulle Nothing concrete, at this time.

Many of the goals of this PR were accomplished by documenting the grammar for image references and getting that into docker engine. We are chasing loose ends down from there.

We don't have concrete plans to follow this up but we will put something together to address the need for aliasing.