Spec changes regarding URIs ($id, $ref, etc) and location semantics

[gregsdennis]

While discussing what to use for the meta-schema URIs, the topic of deriving meaning from URIs and $id arose, namely that URIs, specifically URLs, convey location semantics, and currently those semantics are not honored.

Core section 9.1.2 states

Implementations SHOULD be able to associate arbitrary URIs with an arbitrary schema and/or automatically associate a schema's "$id"-given URI, depending on the trust that the validator has in the schema.

and Core section 8.2.1 states

Note that this URI is an identifier and not necessarily a network locator. In the case of a network-addressable URL, a schema need not be downloadable from its canonical URI.

With schemas being able to declare their own identifiers, and implementations being recommended to associate arbitrary URIs with schemas, any URI locating semantics are short-circuited. The semantics of the URI are meaningless.

Over the years, this has led to great confusion amonst JSON Schema users (and implementors to some degree) in that the natural expectation when attempting to resolve a $ref to a locatable URI (e.g. https://json-schema.org/some-schema) is that the resource should be available at that location but often isn't.

In an effort to alleviate some of this confusion, a few changes to the spec are proposed:

1. Downgrade the text in Core 9.1.2 (above) from a recommendation ("SHOULD") to an optional behavior ("MAY"). This will more greatly recognize that URI resolution mechanisms are an implementation concern. (I'm currently backing down from needing this, but leaving it as a discussion point.)
2. Remove any text regarding implementations defaulting to not accessing the network. (We will need to explore whether this means adding security notes.) The spec is simply silent here; no recommendation.
3. Add text recommending ("SHOULD") implementations honor location semantics of URIs. This still permits implementations to require pre-loading schemas if they don't want to access the network (current behavior).
4. Remove text recommending that all schemas define an $id. We want to encourage that a schema's identifier is the location where it is available: implicit identification over explicit identification.
5. Add text to recommend referencing keywords (e.g. $ref) be used with location-defining URIs.
6. Add text recommending that if $id is supplied, its URI should be a non-locating format (e.g. a URN). (This is still quite useful for bundling and potentially other scenarios.)

There are a few implicit impacts of these changes:

• Location semantics are implicitly defined by the URI, so we don't need to specify them.
• Defining $id becomes somewhat superfluous for most use cases.
• If a schema defines $id with location semantics that differs from its location, it could be referenced by either URI.

There will be significant changes to how the spec defines and uses URIs, but the hope is that we can make these changes without breaking current functionality. Please feel free to list any areas of text that needs to be changed to align with the above, but I'm less interested in actual text changes at this point. Actual text changes should be made in a PR (eventually). For now, we're looking for confirmation that we're headed in the right direction.

This discussion will inform the meta-schema URI discussion linked above.

[Julian]

I'm not sure I fully understand the problem being solved, or whether here all of the above changes are being proposed, or whether the point is to pick some or one of them to solve the problem --

Downgrade the text in Core 9.1.2 (above) from a recommendation ("SHOULD") to an optional behavior ("MAY"). This will more greatly recognize that URI resolution mechanisms are an implementation concern.

This though seems like a significant step backwards to me.

The point of this line is "hey implementers you should allow users of your library a way to associate URIs with their schema even if that schema isn't hosted anywhere". Many implementations do not make this as easy as they should, I suspect because they don't know this line even exists in the spec.

Changing it to MAY will, in my opinion, do nothing to help users, and simply make it even less overt that this should exist in implementations.

Remove any text regarding implementations defaulting to not accessing the network. (We will need to explore whether this means adding security notes.) The spec is simply silent here; no recommendation.

I don't follow what positive change could come out of this either.

I guess overall I don't understand:

With schemas being able to declare their own identifiers, and implementations being recommended to associate arbitrary URIs with schemas, any URI locating semantics are short-circuited. The semantics of the URI are meaningless.

That sounds like a feature not a bug to me (one that's yeah, underimplemented).

Overall inasmuch as I do understand the problem being addressed ("the concept of a URI vs a URL is not intuitive to schema authors who don't read or care about reading URI specs") I don't personally see any of the above improving things. If anything I'd throw in an additional / replacement, if that is the problem we're trying to solve, which is "schema authors: if you use a URI to identify your schema, and if that URI is a URL, then you SHOULD also make the schema retrievable from that URL, and if not, then don't use a URL, use some other kind of URI", and then we of course ourselves should follow that.

[jdesrosiers]

It sounds like your notion of locating is different (broader, maybe) than the "locator" in URL. The "locator" in URL very specifically denotes that the URL identifies a resource in a distinct location.

I disagree. See RFC 3986 Section 1.2.2, specifically the third and fifth paragraph. It says that caches are part of retrieval.

I think the important thing for this discussion is that JSON Schema doesn't care about locating semantics, and it shouldn't. A schema has a URI that it's identified by (whether it self-identifies with $id or is only identified by where it exists is irrelevant), and an implementation has some notion of how to retrieve the appropriate resource for that URI.

But that ☝️ isn't anything that belongs in the spec. What belongs in the spec is that implementations should be able to associate a URI to a resource. That's it.

Since we can't agree on what URI terms apply to this concept, I propose we use another term so we all understand what concept we're talking about. Retrieving from a cache could be called "cache retrieval". Retrieving from arbitrary association could be called "arbitrary retrieval". Retrieving from anything other than cache or arbitrary association could be called "external retrieval". The term "schema retrieval" could be used to refer to the entire process of turning a URI into a schema which encompasses one or more of "cache retrieval", "arbitrary retrieval", and "external retrieval" depending on what an implementation supports.

Ideally, the entire "schema retrieval" process is decoupled from the spec and that concern is delegated to URI and whatever schemes or caches the implementation chooses to support. That's not entirely possible because $id creates arbitrary associations that the spec requires the "schema retrieval" process to take into account. Other than that requirement, I agree that the spec shouldn't impose any constraints on how an implementation retrieves a schema given a URI. It would be awkward for an implementation to support "schema retrieval" in a way that produces different results than the URI scheme semantics would produce, but that's the implementer's choice, the spec shouldn't care.

I'm inclined to agree with @karenetheridge and point (1) in the opening comment about Core 9.1.2 should either stay or become more strict.

It's a very important outcome for me that implementations aren't required or encouraged to support arbitrary association other than what is required by $id and that users aren't encouraged to use arbitrary association (including $id). I'm not saying we need to discourage these things, just don't encourage them. Allowing implementations to not support arbitrary association (with the exception of $id) allows them to encourage the use of scheme based alternatives. For example, to address the use case of schemas that are loaded in memory, implementations can provide a local schema store that uses a custom URI scheme to access it's schemas. This is essentially the same thing we all do with arbitrary association today except you can't use any URI scheme you want.

[gregsdennis]

See RFC 3986 Section 1.2.2

This section doesn't define "locator" or talk specifically about URLs at all, which is what my comment was about.

I'm not sure how this section is relevant other than it says URIs (in general) are merely identifiers, which is what the current JSON Schema spec says. If anything, it seems to confirm the status quo rather than inspire change.

Ideally, the entire "schema retrieval" process is decoupled from the spec and that concern is delegated to URI and whatever schemes or caches the implementation chooses to support. That's not entirely possible because $id creates arbitrary associations that the spec requires the "schema retrieval" process to take into account.

The URI spec section you linked also says that the dereferencing mechanics aren't specified by that document but by other documents (like ours):

Given a URI, a system may attempt to perform a variety of operations on the resource, as might be characterized by words such as "access", "update", "replace", or "find attributes". Such operations are defined by the protocols that make use of URIs, not by this specification. [bold added]

JSON Schema is among the set of "protocols that make use of URIs."

In the context of JSON Schema, arbitrary association is a requirement of resolving URIs because not all URIs convey resolution semantics (because they're only identifiers). If we allow any URI to be used as an identifier, then we must account for the case where something like urn:uuid:9a652678-4616-475d-af12-aca21cfbe06d is used. Since this URI conveys no resolution semantics on its own, we MUST require that implementations allow a URI to be arbitrarily associated with a schema so that resolution of these kinds of URIs can succeed. The only alternative is to restrict users to only using URIs which inherently convey resolution semantics (seems like a step backward).

How an implementation chooses to manifest arbitrary association is irrelevant to the spec. If you want to create a distinct "schema store" and a separate HTTP cache in your implementation, cool, do that. I have a single registry that allows users to register a schema against a URI, and I allow them to define their own external retrieval function. Neither of these approaches are any concern of the spec.

The only thing the spec needs to require of the implementation is that it can produce the appropriate resource when given a URI. To do that, in general, arbitrary association is a requirement.

It's a very important outcome for me that implementations aren't required or encouraged to support arbitrary association other than what is required by $id and that users aren't encouraged to use arbitrary association (including $id)...

I suspect that this is a problem that only you have as no one else has mentioned any issue with it.

Allowing implementations to not support arbitrary association (with the exception of $id) allows them to encourage the use of scheme based alternatives.

Implementations can already "encourage the use of scheme based alternatives," even with arbitrary association. For example, my implementation supports arbitrary association, with or without $id, but that isn't stopping me from creating documentation that says, "Hey, probably use an appropriate URI scheme when writing your schemas."

Furthermore, allowing me to not support arbitrary association doesn't inherently grant me the ability to provide such guidance.

Why are you asserting that one affects the other? What's the link?

[jdesrosiers]

This section doesn't define "locator"

That's fair, but it's never defined what "locator" (or "name") means. The closest thing we have to a definition is that it "provide a means of locating the resource by describing its primary access mechanism". This section also talks about "accessible" resources and "access" operations. Although it doesn't define "access", I took it to be a synonym for "retrieval". I can't imagine what a reasonable distinction would be. Therefore, I think everything this section says about "retrieval" is describing "access" and therefore, it describes how "locators" work.

It's reasonable for you to interpret it differently because the spec isn't clear, but I'm not interested in arguing about what "locator" means. The URI spec doesn't clearly define it and discourages using the term. What I think is relevant is the "retrieval" operation that this section describes. Let's not talk about locators anymore. It really isn't relevant.

JSON Schema is among the set of "protocols that make use of URIs."

I don't think that's what that sentence means. In that sentence, JSON Schema is a component of a "system" that performs operations on a resource. HTTP is a protocol. It defines the operations (GET, POST, PUT, DELETE, etc) that can be performed on resources. JSON Schema is a media type like HTML, not a protocol like HTTP. HTML doesn't define the GET and POST operations it relies on. The operations HTML performs on resources is defined by HTTP.

If we allow any URI to be used as an identifier

The spec allowing any URI to be used is great, but requiring every implementation to support any URI is what I have a problem with. It's totally fine for an implementation to support this, but it should be the implementer's choice what they support.

The only alternative is to restrict users to only using URIs which inherently convey resolution semantics (seems like a step backward).

The spec shouldn't make this restriction, but it also shouldn't restrict implementations from choosing what URI schemes they support and which they don't.

How an implementation chooses to manifest arbitrary association is irrelevant to the spec.

Agreed. I'm still not advocating that this approach has anything to do with the spec. I'm advocating for exactly the opposite. All I want is for the spec to not put limits on how I can implement schema retrieval.

The only thing the spec needs to require of the implementation is that it can produce the appropriate resource when given a URI. To do that, in general, arbitrary association is a requirement.

This assumes that it is/should be a requirement that implementations support any URI scheme. I very much disagree with that. That should be the implementer's choice.

that isn't stopping me from creating documentation that says, "Hey, probably use an appropriate URI scheme when writing your schemas."

I'm not talking about documentation. No one reads documentation. "Encourage" might not have been the best word. I want to be able to make implementation choices that make the right way the only choice users have. That might mean not allowing the use of schemes my implementation doesn't know because I can't enforce the right way to use it. That should be a choice the spec allows me to make.

[gregsdennis]

requiring every implementation to support any URI is what I have a problem with.

If you don't do this, you lose interoperability, as you can't guarantee from one implementation to the next that a URI will be supported.

All I want is for the spec to not put limits on how I can implement schema retrieval.

The current spec doesn't put limits on this. Implementations are free to do schema retrieval however they like. The only thing the spec requires is association between a URI and a schema. That's not putting any limitations on you; it's requiring the implementation to provide an interface to the user. If anything, it's adding capability, not limiting it.

---

Unless I hear a convincing argument from one or more other members, I'm not budging on this. Arbitrary association is staying.

[jdesrosiers]

I'm not budging on this. Arbitrary association is staying.

Can you clarify your position here. I'm a little confused because I'm not arguing for removing arbitrary association, just not encouraging (or discouraging) its implementation. Is it making arbitrary association a MUST that you're not budging on? Or are you ok with leaving it as is (SHOULD) or weakening as proposed (MAY)?

[Relequestual]

I was initially opposed to this direction, however there are some good arguments. On balance and reflection, I'm not convinced it's the right direction, and feels to me like a weakening, which I feel would be bad.

I thought that the WHATWG URL specification removed the ability for a URL to be an identifier only, but it does not.

I agree, there's a general confusion over how references work in relation to URIs, but I think people equally expect it to work in the local file system vs making HTTP requests.

I agree with @karenetheridge here. I think one of the main reasons people struggle to understand how URI resolution works in relation to JSON Schema is due to lack of consistency and documentation across implementations. Mandating that consistency could really help. (I also think that by providing a way for implementations to link to specific parts of their documentation, such as how to associate schemas with URIs, would send a strong signal, and that's one of my reasons for wanting to do so.)

I think preferably, implementations SHOULD also provide an interface to allow for alternative user defined resolution of URI/L to schemas. That might be making an HTTP request. That may well be a solution provided by the same implementation. But I do feel strongly that libraries shouldn't be making HTTP requests elsewhere without a user's specific request/setup.

I agree that you've correctly identified a common challenge, but I think the proposed solution is looking to make it "simpler" but at a cost to what users can reasonably expect. With schemas which have many many references, if implementations default to making HTTP requests, if you're making 30 or 40 requests, that's going to hugely hit performance, especially if it makes that request every time it runs validation against an instance with no cache.
"But someone would never do that!"
The recent traffic analytics data we've gained access to suggests otherwise.

What about the average user? Even a single HTTP request is going to hit performance. We should encourage traversal before processing to verify required referenced scheams are present, and if explicitly enabled, make requests elsewhere to retrieve schemas where possible.

4. Remove text recommending that all schemas define an $id. We want to encourage that a schema's identifier is the location where it is available: implicit identification over explicit identification.

IMHO this would play havoc with schema registries. I feel like there are a number of considerations like registeries you may not be considering here.

I think it comes down to, what do we believe should be the default and what should be (potentailly, if implemented) configurable. I feel that "offline first" is the best practice and most secure approach, while the counter argument is that an "online first" approach better matches user expectations. I'm not convinced changing this to meet newer users expectations is preferable over the performance and security consideraitons.

[jdesrosiers]

I thought that the WHATWG URL specification removed the ability for a URL to be an identifier only, but it does not.

WHATWG shouldn't have anything to do with this, but yes, that's another thing we've been telling people that isn't correct. WHATWG and IETF have done essentially the same thing. They chose one term that encompass all concepts and recommended that people use just the one term. The only significant difference is that WHATWG chose "URL" and IETF chose "URI".

I think the proposed solution is looking to make it "simpler" but at a cost to what users can reasonably expect.

I think it's the other way around. Respecting URI scheme semantics brings JSON Schema behavior in line with how users reasonably expect URIs to work. However, the cost is that using URIs properly is more complex (especially for implementers) than blindly mapping URIs to schemas like we recommend now.

But I do feel strongly that libraries shouldn't be making HTTP requests elsewhere without a user's specific request/setup.

We aren't suggesting that they should. This would continue to be our best-practice recommendation. All we're saying here is that this is an implementation detail that the spec shouldn't be concerned with. All of your arguments about the difficulties with HTTP requests are valid, but they're all just implementation details that implementers need to consider if they choose to support http(s): URIs.

Keep in mind that this proposal doesn't actually change anything about JSON Schema works. It just encourages users to use URIs appropriately and enables implementations to help users make good choices. The expected outcome of this proposal isn't that more implementations will support HTTP requests. The expected outcome is that users stop choosing to use http(s): URIs unless they really intend to host the schema at that URI.

IMHO this would play havoc with schema registries.

Why? How does not using $id affect registries? In any case, $id isn't being removed. It isn't even being deprecated. All this is doing is removing the recommendation the that all schemas use this keyword. It's not even reversing the recommendation. It's just letting users decide if and when they need.

I think it comes down to, what do we believe should be the default and what should be (potentailly, if implemented) configurable.

I disagree. I think that what implementations do by default is an implementation detail that never should have been in the spec in the first place. We can and should provide best-practice guidance, but not as spec requirements. To repeat the point above, remember that the goal of this proposal isn't to get more implementations to support network requests by default. The goal is to get users to stop using http(s): URIs unless they actually intend to host their schemas at those URIs. We want them using better alternatives instead.

[jviotti]

Some of my own thoughts here, not because I'm against the proposal, but to get feedback on my way of thinking about it:

• JSON Schemas are resources, in the same way as an HTML file is a resource, etc (is that even the right term?)
• As it stands, JSON Schema allows schemas to declare an $id that is a valid URI. In my mind, this is similar to HTML files providing a canonical URL to themselves. You might get the HTML file on another URL (given redirection or whatever), but that is the canonical one
• My own "user" understanding of that is that my ID can either be a URL or a pure URI using the URN scheme, etc. This is great
• If I choose to name my schema using a URL, the whole point of a URL is that it is "locatable". Therefore, makes sense that visiting such URL resolves to the schema in question
• If I want to add an ID to a schema, but I don't want to make it locatable, then I should use a URN or something like that. Then the intent is obvious
• As such, using a "fake" URL to name a schema, where such URL is not actually accessible feels very confusing from a user point of view

If the problem is confusing around people using URLs and (rightfully so?) assuming it can be located through the URL, maybe recommend URNs here? That sounds like the correct way of using URIs to me for supporting both cases.

[jviotti]

On the problem of implementations firing lots of HTTP requests, with the security concerns that implies, that sounds like an implementation detail/problem to me.

In my mind, implementations should support some form of caching and/or schema association, but if the user really wants to do HTTP requests, also support that. This is analogous to a traditional file system. You can provide caching and preloading of certain files as optimizations, reach the actual storage when needed, or even do direct I/O and forget about the cache if you really want that.

[jdesrosiers]

JSON Schemas are resources, in the same way as an HTML file is a resource, etc (is that even the right term?)

👍 Yes, "resource" is the right term.

As it stands, JSON Schema allows schemas to declare an $id that is a valid URI. In my mind, this is similar to HTML files providing a canonical URL to themselves. You might get the HTML file on another URL (given redirection or whatever), but that is the canonical one

That's not a bad analogy, but the concepts work a bit differently from one another. A canonical link tells you that there's another URI that maps to this resource. $id creates a mapping from the URI to this resource and evaluation continues as if you got the schema using that $id even if you actually got it from some other URI.

There's no perfect analogy for $id in other web technologies. It's unique (and I don't think that's a good thing).

My own "user" understanding of that is that my ID can either be a URL or a pure URI using the URN scheme, etc. This is great

This is something we've been telling people, but I'm arguing that it's wrong. Any URI that maps to resource becomes a "locator" even if the semantics of the URI scheme say it's not intended to be a locator. The schema is located by using the mapping you defined. If you choose to a URI scheme that isn't intended to be used as a locator, you're using that URI scheme incorrectly by mapping it to your schema. The only concept in JSON Schema that uses URIs as pure identifiers is vocabulary URIs. An example in another web technology is custom link relations, if you're familiar with those.

If I choose to name my schema using a URL, the whole point of a URL is that it is "locatable". Therefore, makes sense that visiting such URL resolves to the schema in question

👍

If I want to add an ID to a schema, but I don't want to make it locatable, then I should use a URN or something like that. Then the intent is obvious

This is still technically wrong, but less confusing that contradicting how the URI scheme says the URI should map to a schema. Remember the URN becomes a locator because you mapped it to a schema.

As such, using a "fake" URL to name a schema, where such URL is not actually accessible feels very confusing from a user point of view

👍

If the problem is confusing around people using URLs and (rightfully so?) assuming it can be located through the URL, maybe recommend URNs here? That sounds like the correct way of using URIs to me for supporting both cases.

That's essentially where Greg and I are headed with this proposal. URNs aren't a good fit for reasons that aren't important to get into in this discussion. If this proposal is adopted, we're going to recommend a custom schema: URI scheme be introduced that we would expect people to use anywhere they're currently using http(s): for something that isn't hosted.

[jviotti]

Thanks for the thoughtful response! Makes sense and I like the direction of the proposal.

[Relequestual]

If this proposal is adopted, we're going to recommend a custom schema: URI scheme be introduced that we would expect people to use anywhere they're currently using http(s): for something that isn't hosted.

That dramatically negates some of my objections. I have less reservations in that case.

[ssilverman]

I’m with @gregsdennis on all this. Here’s a quick summary of the definitions: URI’s are opaque IDs having a specific format. URL’s essentially assign semantics to the scheme part of a URI and the contents. URI’s are a superset of URL’s.

Just looking at a URI does not tell you whether it’s supposed to be a URL or URI. It’s up to a specification and context. This means if a specification says to use a “URI”, then it doesn’t matter if “http” is the scheme or if it’s confusing to a user. URI’s are essentially opaque ID’s. That’s it.

JSON schema use URI’s which I think are the correct choice. I also disagree with catering to people who “don’t read the documentation.” (Mentioned in one of the comments above.) I also disagree with forcing a specific URI scheme or that users should not use so-called “confusing” URI forms such as “http.”

This is an infrastructure-type specification and, in my view, it’s inappropriate to mandate what URI’s look like. That is, user-supplied ones specifically. The specification authors get to decide whatever they like for well-defined ones that aren’t supplied by a user.

[I decided to make this a top-level comment instead of a reply because there’s lots of stuff and wanted to share my thoughts.]

[ssilverman]

For context, the last comment I saw was this one: https://github.com/orgs/json-schema-org/discussions/460#discussioncomment-6696131

[jdesrosiers]

It's good to hear from you Shawn.

[ssilverman]

Thank you. That’s nice to hear. :)

[awwright]

I think adding some guidance about using tag: URIs may be helpful, but the current usage of the URIs as identifiers, which only potentially (and not necessarily) have a machine readable target, is the correct behavior here. This is the same design of XML, and this article lists some of the problems I've been trying to avoid: https://www.w3.org/blog/2008/w3c-s-excessive-dtd-traffic/

URIs are often misunderstood because Web browsers use a very specific subset of what they're capable of. But not every misunderstanding like this can be addressed in a specification, adding implementation guidance can only do so much.

Downgrade the text in Core 9.1.2 (above) from a recommendation ("SHOULD") to an optional behavior ("MAY"). This will more greatly recognize that URI resolution mechanisms are an implementation concern.

The passage in question ("Implementations SHOULD be able to associate arbitrary URIs with an arbitrary schema") is saying there has to be some way to map a URI to a schema. This is required to use $ref to do recursion; the only reason this isn't a MUST is because that would be slightly out of scope; there might be legit reasons for an implementation to do it some different way.

Remove any text regarding implementations defaulting to not accessing the network. (We will need to explore whether this means adding security notes.) The spec is simply silent here; no recommendation.

This passage is reenforcing the usage of URIs as an identifier. The only reason this isn't a "MUST" is to support the use of hypermedia and network distribution. If your application is a hypermedia application then use the network, but on the other hand, you shouldn't adopt network usage of JSON Schema if your application doesn't do anything with hypermedia.

Add text recommending ("SHOULD") implementations honor location semantics of URIs. This still permits implementations to require pre-loading schemas if they don't want to access the network (current behavior).

The meaning of URIs in network location (i.e. URLs) is already defined in the normative references (RFC 3986), so adding a "SHOULD" would be redundant.

Remove text recommending that all schemas define an $id. We want to encourage that a schema's identifier is the location where it is available: implicit identification over explicit identification.

There's only a few situations where this is beneficial, in most cases it could only cause confusion. The point of having the URI in the document is so you can take the document with you anywhere, and it means the same thing. That's easier if the document gives itself a URI, rather than supplying its Base URI out-of-band.

Sometimes you do want the base URI to change when you move the document around... that's why $id is optional. It's just not very common.

Add text to recommend referencing keywords (e.g. $ref) be used with location-defining URIs.

To rephrase the principle: Unless the application is already a network application, the act of validating JSON with a schema should not depend on a network request, or the network being up at all.

Add text recommending that if $id is supplied, its URI should be a non-locating format (e.g. a URN). (This is still quite useful for bundling and potentially other scenarios.)

Indeed, picking a URN for "non-information resources" is an under-used technique, we could add some implementation guidance to use a urn: or a tag: URI.

[jdesrosiers]

I don't think urn: is appropriate for users to use in most cases. URNs are supposed to be globally unique. One way this is ensured is that you're supposed to register your namespace and define rules for assigning identifiers. If we encourage users to use urn:, if they are using them properly (which they probably won't), they would need to register their namespace, which would overwhelm the registry which wasn't designed for every project that uses JSON Schema to have a registered namespace.

URNs are also supposed to be forever. They always refer to the same thing. Most schemas evolve which would require people to change their identifiers every time the schema changes. That's generally good practice, but not always feasible as we've seen with our meta-schemas.

These are the reasons the tag: scheme was introduced.

[jdesrosiers]

this article lists some of the problems I've been trying to avoid: https://www.w3.org/blog/2008/w3c-s-excessive-dtd-traffic/

I don't think that's relevant. It's about misbehaving clients. Misbehaving clients should not drive the architecture of JSON Schema. There will always be misbehaving clients. Our current spec strongly discourages network access, yet we still have the same problem this article describes with our meta-schemas. If discouraging network access was supposed to protect us from this situation, it failed.

[awwright]

That's currently only the case with vocabulary URIs. All other URIs used in JSON Schema do necessarily have a machine readable target. That target is a JSON Schema document. An implementation can't function without retrieving that JSON Schema and evaluating it

OK yes, it would have been more accurate to say that the URIs are not necessarily network-dereferencable, even if the schemas they point to are machine-readable.

That said, it's not strictly necessary that the target has to be machine readable either, there's no technical reason $ref couldn't point to a resource besides a JSON Schema, so long as it follows the same semantics (provides validation errors and annotations). You'll just limit the kinds of validators that read those schemas. (This isn't much different than observing HTTP servers don't have to serve HTML, you're just limiting the audience who will be able to read the response.)

---

How exactly is this the same design as XML?

This is described in the blog post I linked to:

Note that these are not hyperlinks; these URIs are used for identification. This is a machine-readable way to say "this is HTML". In particular, software does not usually need to fetch these resources, and certainly does not need to fetch the same one over and over!

I don't know if there's anywhere else that puts it nearly as nicely. XML and XML Namespaces have little guidance on this, they mostly imply infrequent network access by the fact that if the network were required, it would say so (specifically, to dereference). PUBLIC identifiers are supposed to be identifiers inherently known to the XML parser; SYSTEM identifiers are one way that permits a network dereference, but even then it's optional:

An XML processor attempting to retrieve the entity's content may use any combination of the public and system identifiers as well as additional information outside the scope of this specification to try to generate an alternative URI reference.

That is to say, if the parser knows about the URI but not its contents, it may provide the contents of the reference from any location, like an alternate location. https://www.w3.org/TR/xhtml1/dtds.html specifically suggests that the DTDs should be provided locally, and provides copies to bundle with software.

I think the fact XML has been so terse on this is has led to the many problems described in the blog post, there's a fair number of editors who eschew implementation guidance in specifications (Idk about the editors on XML, though, but the implementation guidance there also seems pretty sparse).

---

As long as all of your schema references use relative URIs, you can still take your schemas anywhere and it means the same thing without the need of $ids

I agree file: URIs are under-used, and this should definitely be possible. But if you want to share a schema where multiple parties can refer to the same schema, you need a URI that everyone can agree to re-use, and file: URIs are difficult to share like this; although they support hostnames (file:///etc/passwd is an alias for file://localhost/etc/passwd) they're virtually never used like that.

Also, $id is most significant for the URI-base-changing behavior that <base href="foo" /> in HTML provides, it shouldn't be so foreign a concept to people.

[jdesrosiers]

there's no technical reason $ref couldn't point to a resource besides a JSON Schema

Of course not. I'm not saying references can only point to schemas. The distinction I was making was between URIs that an implementation needs to dereference (not necessarily on the network) in order to function and URIs that don't.

This is described in the blog post I linked to:

Note that these are not hyperlinks; these URIs are used for identification. This is a machine-readable way to say "this is HTML". In particular, software does not usually need to fetch these resources, and certainly does not need to fetch the same one over and over!

I don't think that has any relevance to what we're discussing here. This is equivalent to if implementations were repeatedly trying to download a resource from a vocabulary URI. Trying to dereference that URI doesn't even make sense. There's nothing the implementation should expect or need from that dereference.

Referencing a schema is a completely different case. Those URIs do need to be dereferenced. There is something that URI refers to that the implementation needs to do its job. A dereferencing operation doesn't necessarily need to hit a network. The resource could come from a cache. But, JSON Schema is alone in that it isn't the default expectation that when URIs are dereferenced, they are dereferenced in a way appropriate to its scheme and naming authority.

if you want to share a schema where multiple parties can refer to the same schema, you need a URI that everyone can agree to re-use, and file: URIs are difficult to share like this

This approach isn't limited to the filesystem. You can drop the same schemas on a web server and what ever URI is used to serve those files is the common URI everyone can use.

[jdesrosiers]

Thanks for providing the XML references. Form reading a bit about external entities and SYSTEM vs PUBLIC identifiers, my understanding is that PUBLIC identifiers are similar to the arbitrary association used in JSON Schema. However, my impression is that they are supposed to be common, well-known identities that are expected to be the same globally. A good parallel is meta-schemas, but not much else falls into that category. SYSTEM identifiers seem to be more like what I've been talking about where the URI is expected to refer to a filesystem or network location that is used to retrieve the resource.

XML doesn't suggest one mechanism be used over another. While XML expects PUBLIC identifiers to be mapped according to some internal mechanism, it also expects and allows for implementations to retrieve resources on filesystems and the network. I'm advocating for the same thing XML does, allow and expect either approach and let users and implementations use what makes most sense for them and their use cases.

SYSTEM identifiers are one way that permits a network dereference, but even then it's optional

I don't think calling it "optional" is representative of what the spec says. It appears to me that XML decouples the responsibility of how dereferencing a URI is performed. It's "optional" only in the sense that it's not specifically mentioned. That also doesn't mean it's discouraged. This is all in line with what I'm advocating for for JSON Schema, don't encourage or discourage any particular approach and allow implementations and users to do what they need.

[Julian]

I don't have much to add that hasn't already been said elsewhere.

But just to "officially note" what I think my opinion is having watched this week's Monday call (which I think isn't on YouTube yet so can't link to it for those who don't have context, though I'm sure will be at some point), will leave this below.

I would personally:

• Not change the SHOULD to MUST, because I agree that if an implementer thinks arbitrary identification of any URI is confusing they should be free to not implement that functionality
• Not remove or weaken the clause either, because:
  1. it's been there historically, which is always an argument to keep it (i.e. the bar should be way higher to change or remove specified behavior than it is to add)
  2. I do probably tend to agree in practice that it's not arbitrary identification that's confusing to users, so I personally do think it's a good recommendation. Of course again that doesn't mean someone shouldn't prove it wrong. But I think the world would be better if more implementations followed this SHOULD -- not enough implementations support this behavior currently. I maintain this is simply because many implementers don't know this clause is here, or don't understand what it means!
• [EDIT after discussion below]: Be neutral on moving network retrieval restrictions from the spec elsewhere to some best practices docs
• But I would possibly consider that it might be a decent compromise to simply add an additional clause saying "implementations MAY limit which schemes may be used in this way" or something, or equivalently given the current language, just modify the word "arbitrary URIs" to be a bit more liberal and simultaneously say implementations should allow associating from some pool of arbitrary URIs (probably by scheme) but not necessarily any URI on the planet.
• And finally I would not personally be particularly in favor of mentioning or recommending urn, tag, or a custom scheme, as the semantics of those I think are even more confusing to users, half of whom have never seen one before, a quarter of whom wouldn't know that they then couldn't use relative references with them, and the other quarter are already asleep thinking about this.

TL;DR: my opinion I think is (still?) "everything should stay as-is", though maybe adding another sentence or tweaking the language makes everyone happy.

My 0.02 :D

[Julian]

Embedded schemas is certainly a place where non-locating URIs makes sense, but writing compound schema documents by hand has always been discouraged. It's intended to be the result of bundling several schema documents.

Can you share where we discourage this? I suspect the majority of users who do know it exist do this that I've seen. I certainly have personally many times, so would love to know if it's discouraged, it's way more convenient than alternatives when you do the "usual" evolution of a simpler schema into a more complex one.

[jviotti]

I want to ensure that JSON Schema isn't over constrained in ways that block or discourage novel usage or applications.

I personally think this comment by @jdesrosiers 's touches on the core of it. Seems like a lot of disagreement in this thread is because all of us use JSON Schema in different contexts, in different ways, for different purposes (which is a great thing!), and its hard for specific recommendations to be universally applicable to all of these cases.

Maybe leaving recommendations out of the spec makes sense. Instead having guidelines that evolve within specific contexts and use cases in mind?

As an analogy, consider the development of a general purpose programming language like C++. Different areas, like embedded, real-time, app development, etc use, promote, and even discourage different aspects of the language. That doesn't mean some of its features are bad. They often have their use cases in specific areas. Even goto (a highly discouraged keyword) has true legit use cases where there is no practical replacement. If C++ was too restrictive in the name of "protecting users", it would do itself more harm than good.

[jdesrosiers]

Can you share where we discourage this?

I could have sworn the spec said this, but reading it again, all it really does is acknowledge that both bundling and manual approaches exist. UJS says that it's unusual to bundle manually. I think that's true, but I guess I could be wrong. I personally discourage it because it confuses people and it's much more straightforward to just make a separate schema. I also haven't encountered a case where I thought an embedded schema was a better choice than separate schema, but maybe that just speaks to my style choices.

[gregsdennis]

I'm just catching up with this, and I'm about halfway through what was posted overnight (for me), but here are a couple immediate thoughts.

Joe should use a URN or tag: and not a URL then. - @jviotti

Yes, but URNs and TAGs aren't the only non-locatable URIs. Furthermore, this conversation started by recommending we use a schema: scheme, which we get to define. We can define it to mean that the resource has no accessible location (in the sense that an https: URI is locatable) but can still have relative semantics for the purpose of building new URIs.

People will read the spec and learn that JSON Schema isn't intended to work on the network and therefore not appropriate for use with API clients - @jdesrosiers

We've had this recommendation in the spec since as long as I've been part of the project, and I can't say I've ever heard people questioning whether it's okay to use JSON Schema in a networked application. People just use it on a network anyway because implementations just support it anyway. If anything, the recommendation to avoid network access is simply ineffective.

For me, by having this offline recommendation, we alleviate ourselves from having to specify security concerns. It offloads those concerns to implementations, which (if they're anything like mine) offload those concerns to the user. By including this statement, we're requiring that the end user be responsible in making the judgement to go online.

[gregsdennis]

I think it discourages the use of non-locating URIs, which I think is a good thing. - @jdesrosiers

I disagree that discouraging the use of non-locaing URIs is a good thing. In fact, we're actively in the process of trying to determine if we should start using a non-locating URI for the live meta-schema, so I don't understand this comment at all.

Nearly all JSON Schema usage is people working with files on a filesystem.

I haven't seen evidence of this. This is a pretty bold claim about the entirety of the JSON Schema ecosystem; one that I don't think any of us are qualified to make.

Your position is understandable if this is what you believe, but we shouldn't make assumptions like this. We need to consider that schema usage can be and is used in many contexts.

Embedded schemas is certainly a place where non-locating URIs makes sense, but writing compound schema documents by hand has always been discouraged. It's intended to be the result of bundling several schema documents.

I agree with @Julian that this is not something that we've discouraged. Maybe you have, but as a group, and through the specification, we certainly have not discouraged people from hand-composing schemas for any purpose. Quite the opposite: we have actively discouraged the automatic generation of schemas (e.g. from code) as such generation cannot adequately support the nuances of modern JSON Schema and generally needs the touch of a human before it's production-ready.

UJS says that it's unusual to bundle manually.

I agree that it may be unusual to bundle manually, but that's not really a discouragement. It's just implying that bundling tooling exists and should be used. It's a fine line between a recommendation for one method and a recommendation against another. Splitting hairs, probably.

Still, it says nothing about authoring compound schemas, which the spec does cover in that it recommends that schemas have $id defined. There are even examples outside of the context of bundling. I'd say this is pretty far from a discouragement.

Even goto (a highly discouraged keyword) has true legit use cases where there is no practical replacement.

The discouragement is there to provide guidance that alternate means should be sought out and that using this keyword is to be a last resort.

Similarly, a JSON Schema implementation SHOULD provide means to operate offline. Online operation incurs risks which offline operation alleviates. Not only security concerns, but availability and other concerns as well.

If I'm happily validating data against schemas in an online environment, and my internet connection drops, then suddenly schemas that previously worked no longer do. Requiring that users built infrastructure to handle such occurrences places a significant burden on them. A recommendation that implementations support offline scenarios is a much simpler solution for the user (and it's not hard to implement).

[gregsdennis]

I think it's time to wrap this up. Going around arguing our points isn't reaching conclusions, so here's where I think everyone is:

It seems that the majority is in favor of the status quo. While they may be open to some tweaks to the language, there should be no changes to the overall requirements or recommendations as these recommendations serve a purpose (as outlined in the discussion above). This majority includes myself, @Julian, @awwright, and @karenetheridge (edit: and @ssilverman).

In favor of removing some of the recommendations is primarily @jdesrosiers with a little support from @jviotti (the only participant not part of the core team).

As such, I propose the following (specific text TBD):

1. Arbitrary association is still a SHOULD requirement. Implementations still have the choice to not support this, but there are legitimate reasons for the spec recommending this feature.
2. Relax (or remove) the recommendation to avoid accessing the network. Potentially, this could be reworded to encourage a default "offline" mode rather than discouraging an available (default off) "online" mode.
3. The spec will maintain that URIs are merely opaque identifiers. We would also
   • provide guidance in supplemental literature (e.g. UJS) that if URLs are used, the schema should be available at that location.
   • lead by example by using a non-locating URI for the non-locatable live meta-schema.
4. Keep the recommendation that schemas define $id. We may add text that aligns its usage with point (3) above.
5. No changes to recommendations of URIs found in $refs.
6. No recommendation (at least in the spec) that $id should be a non-locating URI. Maybe this could be in supplemental literature as a best practice with examples and reasoning, but the spec should be non-prescriptive.

Overall, the only changes are some wording tweaks in the areas of (2) and (3). Again, no policy changes.

[Relequestual]

I'm talking about supporting arbitrary association, but only allowing (or not allowing) specific schemes to be used. I should be able to reject a schema that uses a file: URI in $id, while still allowing other schemes to be used with $id. Not only should I be able to do that, it's very important to me that that's not a choice that the spec discourages.

I've been thinking about this considerably over the course of a few days. I very strongly disagree.

Consider, a developer wants to have a separation of concern with loading content from a file system or making http requests, and running validaiton.

If you prohibited arbitrary association for say file:, the developer could not do all the file system actions initially, and then proceed to use in memory only representations of schemas which use file:. Why do this? Well, they could have validation pushed out to a lambda function or scaled to a cluster manually for large jobs of big data. These are both things I have seen several orgs do. (This also applies to http/s:.)

It's unreasonable to expect them change all their schemas $ids and $refs to allow them to work offline/without a file system.

The spec language as it stands allows you to make such a limitation, but if many implementations did this, it would weaken and fragment the ecosystem.

We've previously discussed "well, we could have a local cache". Great, yes, we sure can, but why not allow the developer to SUPPLY that rather than the implementation requiring that only the internals of the implementation can build that?

Maybe some users of these schemas DO want to use them manually, using filesystem or http requests. They shouldn't be hobbled. There are multiple use cases that a schema author should be able to support concurrently, which includes those when used by third parties with different constraints.

What is the harm?

Given the above, I conclude such a change would, critically, prevent schema authors from being able to empower others. I feel this outweighs "people who have gotten confused trying to use both because they don't understand what they're doing."

I've answered enough support questions from exactly this situation to come to the conclusion that the best thing is for the implementation not to offer both.

We need better documentation, not to limit legitimate and in-the-wild use cases.

Having said all that, if you want to make your implementation's users jump through additional hoops for use cases you consider erronious or problematic, then that is your prerogative.

I don't think it's problematic to make schema authors aware of the problems, but I don't think the spec making suggestions that it's OK for implementations to prevent some forms of arbitrary association is going to do that.

[jviotti]

My take on this is:

1. As @gregsdennis and @jdesrosiers said, arbitrary URI <-> schema association will probably always be there, as there are use cases that demand it. Most implementations already support it (see addSchema in Hyperjump)

2. I think what @gregsdennis is proposing is that users of JSON Schema will very likely not read the spec, and the spec is probably not the place for opinions or best practice advice. We can have i.e. UJS and other docs recommend that URI semantics are obeyed (i.e. if you use a URL, then try to serve the schema at that URL indeed, and same for file:). Of course, disobeying these best practices is still technically valid

3. Assuming point 1, implementations should not prevent file: to be used for arbitrary association that doesn't really perform I/O. We already let users use arbitrary association with i.e. http: URLs without actually resolving to such URL, so it follows that we should allow people to use file: without I/O access if they want to. I agree with @jdesrosiers 's that this is confusing, but using an http: URL that doesn't really resolve to anything is equally confusing IMO

I should be able to reject a schema that uses a file: URI in $id, while still allowing other schemes to be used with $id. Not only should I be able to do that, it's very important to me that that's not a choice that the spec discourages.

The question here is whether implementations are allowed to enforce best practices or opinions, or not. I agree with @jdesrosiers that if the spec says you should always support arbitrary association, then making an implementation that enforces a stricter mode maybe feels like not 100% compliant anymore, therefore such best practice or opinion is discouraged?

I wonder if you can achieve both though. I.e. make Hyperjump enforce some best practices, while still allow a "force" mode that still lets the user use whatever $id they want? It could be more subtle though. For example, let the user define the set of protocols they want to use, but defaulting them to a strict set. Then, it is an implementation configuration problem with non-divergent stricter defaults?

[Relequestual]

I wonder if you can achieve both though. I.e. make Hyperjump enforce some best practices, while still allow a "force" mode that still lets the user use whatever $id they want? It could be more subtle though. For example, let the user define the set of protocols they want to use, but defaulting them to a strict set. Then, it is an implementation configuration problem with non-divergent stricter defaults?

I'd find this acceptable. It would not modify the specification requirements.

[Julian]

For what it's worth, a large portion why I have no issue with adding the MAY line I proposed above is because:

The spec language as it stands allows you to make such a limitation, but if many implementations did this, it would weaken and fragment the ecosystem.

I suspect no one will indeed do this other than Jason :) -- many implementations, as we've said many times, don't do what's already there (so there's clearly the separate problem to fix).

I agree with @jdesrosiers that if the spec says you should always support arbitrary association, then making an implementation that enforces a stricter mode maybe feels like not 100% compliant anymore, therefore such best practice or opinion is discouraged?

I think it seems most of us think (regardless of the spec text) that not supporting arbitrary association in full is discouraged / isn't a best practice.

That being said I think he should do it and prove that wrong if he's convinced! All we seem to continue to be arguing about vis a vis this specific change, is about is whether the SHOULD in the spec is enough to say there's liberty to do so or not.

[jdesrosiers]

@Relequestual

If you prohibited arbitrary association for say file:, the developer could not do all the file system actions initially, and then proceed to use in memory only representations of schemas which use file:. Why do this? Well, they could have validation pushed out to a lambda function or scaled to a cluster manually for large jobs of big data.

I don't think I understand the scenario you're describing here. It sounds like you're saying the developer wants to work with schemas as files in one space, and also wants to use use those same schemas in a lambda. I don't see how an implementation disallowing using the file: scheme for arbitrary association impacts that situation.

If you're using schemas as files, you don't have $ids and your $refs are filesystem-relative. In practice, you don't have $ids because it pins those schemas to work only on your filesystem and exposes your filesystem details to anyone you share the schema with, which is a big security no-no. Not allowing file: schemes protects users from this security misstep.

In practice, $refs are file-system relative because it would be extremely inconvenient to use full paths and pins the schemas to a single filesystem and exposes your filesystem details to anyone you share your schemas with. Implementations should protect against using full paths here as well. In both the case of $id and $ref, the restrictions only serve to protect users and they're extremely unlikely to encounter those cases anyway because there was no good reason to do so in the first place.

So, now you want to use your schemas in a lambda function. No problem. They'll work exactly the same way. You don't need to give them $ids or otherwise associate them with URIs. Their identifiers are determined by their path on the filesystem and the references all just work because they are already filesystem relative. So, disallowing arbitrary associations file: doesn't stop you from doing anything you might want to do with your schemas. It only protects you from exposing your file system details.

If, for some reason, you wanted to make things a little harder on yourself and load those schemas into memory, you can do that with your $id-less, relative-$ref schemas nearly as easily as you would with self-identifying schemas. With self-identifying schemas, you still need to load the schema from the file it's stored in, so the only difference is that you need to assign the $id-less schemas a URI when you load them. You can trivially create these URIs automatically by applying a transform on the path to turn it into a URI using the scheme of your choice, as long as the scheme supports paths and isn't file:.

We need better documentation, not to limit legitimate and in-the-wild use cases.

I hope I've shown that schema authors are not being limited in any way by not allowing them to use file: URIs for arbitrary association. It's just protecting them from doing something that could harm them that they almost surely weren't doing anyway because it's so inconvenient and limiting to do so in the first place.

@jviotti

2. I think what @gregsdennis is proposing is that users of JSON Schema will very likely not read the spec, and the spec is probably not the place for opinions or best practice advice. We can have i.e. UJS and other docs recommend that URI semantics are obeyed (i.e. if you use a URL, then try to serve the schema at that URL indeed, and same for file:). Of course, disobeying these best practices is still technically valid

I think you meant that's what I was proposing. Greg is arguing for keeping these recommendations in the spec. It's clear that there's strong support for keeping these recommendations. At this point I'm just trying to ensure that these recommendation don't overly constrain the spec.

We already let users use arbitrary association with i.e. http: URLs without actually resolving to such URL, so it follows that we should allow people to use file: without I/O access if they want to.

They're similar, yes, but I think file: is a far more extreme case because of the security implications of exposing filesystem details. But, even just from the "it's confusing" perspective, I think it's even more confusing for the filesystem than network requests because the filesystem is closer to the user.

Given JSON Schema's history with using and promoting http(s): URIs, I would probably choose to emit a warning when users use http(s): for arbitrary association and suggest they use something else. However, I strongly believe that file: URIs should be prohibited for implementations that support file-based schemas for security reasons.

let the user define the set of protocols they want to use, but defaulting them to a strict set. Then, it is an implementation configuration problem with non-divergent stricter defaults?

This is what I have in mind.

I'd find this acceptable. It would not modify the specification requirements. - @Relequestual

That's great, but I still think the spec discourages that solution.

@Julian

I think it seems most of us think (regardless of the spec text) that not supporting arbitrary association in full is discouraged / isn't a best practice.

I agree. It's clear now that most of you think that although it should be technically allowed to limit what schemes can be used for arbitrary association, you also think it should be discouraged and that discouragement belongs in the spec. Thank you all for expressing your opinions on this point so it's not just me and Greg going back an forth. I'm still going to do what I think is best and safest for my implementation, but I'm not going to argue this point any more.

[gregsdennis]

Proposal: json-schema-org/json-schema-spec#1435

[gregsdennis]

PR accepted. Closing this discussion.

[gregsdennis]

Also related: json-schema-org/json-schema-spec#1231