Remove $dynamicRef bookending requirement

[jdesrosiers]

The current process for resolving a $dynamicRef requires that both ends of the dynamic scope have a $dynamicAnchor. This bookending requirement complicates $dynamicRef resolution, makes some use cases impossible, and doesn't seem to serve any useful purpose as far as I can tell.

The only explanation for the bookending requirement in the spec comes from this cref.

Requiring both the initial and final URI fragment to be defined
by "$dynamicAnchor" ensures that the more common "$anchor"
never unexpectedly changes the dynamic resolution process
due to a naming conflict across resources. Users of
"$dynamicAnchor" are expected to be aware of the possibility
of such name collisions, while users of "$anchor" are not.

But, that doesn't make sense. I don't see how $anchor could effect the dynamic resolution process nor what effect bookending could have in stabilizing the process.

I'll follow up shortly with a couple examples of how the bookending requirement is harmful.

(This was originally reported in #1030 and split off here)

[jdesrosiers]

Here's an example of a non-recursive use for $dynamicAnchor/$dynamicRef where it doesn't make sense to require a $dynamicAnchor at the initially resolved starting point.

Hyper-schemers will recognize this pattern. Let's say I have several resources and for each resource, I want two schemas: one for a single instance and another for a list of instances. The list of instances follows a common pattern that looks something like this.

{
  "$id": "https://json-schema.hyperjump.io/schema/list",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "type": "object",
  "properties": {
    "list": { "type": "array" },
    "nextPage": { "type": "integer" },
    "previousPage": { "type": "integer" },
    "perPage": { "type": "integer" },
    "page": { "type": "integer" }
  }
}

I can then compose this schema with another to create a list schema for a specific resource.

{
  "$id": "https://json-schema.hyperjump.io/schema/foo-list",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "$ref": "/schema/list",

  "properties": {
    "list": {
      "items": { "$ref": "/schema/foo" }
    }
  }
}

That's not too bad, but it does couple this schema with the "list" property name. If I decided I want to change that name to "itemList", I would have to change not only /schema/list, but also every schema I composed it with. That's where $dynamicAnchor/$dynamicRef can help.

{
  "$id": "https://json-schema.hyperjump.io/schema/list",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "type": "object",
  "properties": {
    "list": {
      "type": "array",
      "items": { "$dynamicRef": "#list" }
    },
    "nextPage": { "type": "integer" },
    "previousPage": { "type": "integer" },
    "perPage": { "type": "integer" },
    "page": { "type": "integer" }
  }
}

{
  "$id": "https://json-schema.hyperjump.io/schema/foo-list",
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "$ref": "/schema/list",

  "$defs": {
    "foo": {
      "$dynamicAnchor": "list",
      "$ref": "/schema/foo"
    }
  }
}

It's a little awkward, but it's decoupled. I can now change whatever I want about the /schema/list schema and not have to change any of my /schema/*-list schemas.

However, this doesn't work yet because it doesn't fulfill the bookending requirement. According to the way the spec is written, /schema/list should have a $dynamicAnchor somewhere if it's $dynamicRef is allowed to resolve against the dynamic scope. But, there's nowhere it makes sense to add a $dynamicAnchor in this case. You could add it in a definition ...

"$defs": {
  "list": { "$dynamicAnchor": "list" }
}

but that's just putting the $dynamicAnchor somewhere it won't hurt anything just because it has to be somewhere. It's not needed to get the functionality we wanted. It also makes it impossible to make /schema/list abstract in the sense that it can't be used by itself. It can only be used when combined with a schema that sets "$dynamicAnchor": "list".

[jdesrosiers]

Here's what the hyper-schema related meta-schemas might look like if there was 2020-12 hyper-schema release. I've truncated them down to only what is necessary for this demo.

{
  "$id": "https://json-schema.org/draft/2020-12/hyper-schema",
  "$dynamicAnchor": "meta",
  "allOf": [
    { "$ref": "https://json-schema.org/draft/2020-12/schema" },
    { "$ref": "https://json-schema.org/draft/2020-12/meta/hyper-schema"}
  ]
}

{
  "$id": "https://json-schema.org/draft/2020-12/schema",
  "$dynamicAnchor": "meta"
}

{
  "$id": "https://json-schema.org/draft/2020-12/meta/hyper-schema",
  "$dynamicAnchor": "meta",
  "properties": {
    "links": {
      "items": { "$ref": "#/$defs/links" }
    }
  },
  "$defs": {
    "links": {
      "properties": {
        "submissionSchema": { "$dynamicRef": "#meta" }
      }
    }
  }
}

So far there's nothing surprising happening. Now, let's say we want to split #/$defs/links into it's own schema.

{
  "$id": "https://json-schema.org/draft/2020-12/meta/hyper-schema",
  "$dynamicAnchor": "meta",
  "properties": {
    "links": {
      "type": "array",
      "items": { "$ref": "/draft/2020-12/links" }
    }
  }
}

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "#meta" }
  }
}

Without the bookending requirement, we'd be done and happy. But, because of the bookending requirement, we need to jump through some hoops to make this work. We need to use the absolute-URI portion of the URI to change the starting point to a schema that has a "$dynamicAnchor": "meta".

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "/draft/2020-12/hyper-schema#meta" }
  }
}

The $dynamicRef is first resolved like a $ref giving us https://json-schema.org/draft/2020-12/hyper-schema#meta as the initially resolved target schema. Since /draft/2020-12/hyper-schema has "$dynamicAnchor": "meta", the $dynamicRef is now allowed to resolve using the dynamic scope just like it did when it was in $defs in the /draft/2020-12/hyper-schema.

[jdesrosiers]

I've pointed out that if we remove the bookending requirement, there's never a need for a non-fragment-only URI. @ssilverman pointed out there's no reason for it to be a URI at all any more. I think that's exactly the direction this should be going. I think the default behavior of falling back to behaving like a $ref was a reasonable behavior for $recursiveRef because it was only used for making recursive references, but in non-recursive use cases, $dynamicRef behaving like a $ref is just confusing and never what you meant. Not to mention, it opens the door for people to use $dynamicRef in ways that are not even trying to be dynamic leading to confusion. We have already seen this happen.

It would be best if $dynamicRef never falls back to non-dynamic behavior. If there is a reference to a non-existent dynamic anchor, then it should result in an error. Furthermore, there's no use in the $dynamicRef masquerading as a URI. Everything other than the fragment gets ignored in any case where the $dynamicRef is used properly, so we might as well just use the anchor value.

[marksparkza]

I agree wholeheartedly with your argument. I would only suggest that if a $dynamicRef does not resolve to any $dynamicAnchor, it should just pass, as if it were an empty schema. Otherwise (without requiring a bookend) an extensible schema could not be used on its own.

[jdesrosiers]

I would only suggest that if a $dynamicRef does not resolve to any $dynamicAnchor, it should just pass, as if it were an empty schema.

I disagree. If I try to reference something (static or dynamic) that doesn't exist, I expect that to be an error. Imagine the case where I typo a dynamic reference. It would be surprising and would mask the error if it just continues on with no indication of why the schema I thought I was referencing seems to be getting ignored.

Otherwise (without requiring a bookend) an extensible schema could not be used on its own.

It's a feature to be able to create an extensible schema that can't be used on it's own. It's like declaring an abstract class. There are good reasons to want to do that sometimes. And there's nothing stopping you from declaring a default, empty schema the same way I did in the first example. You'd just be doing it because you want a default, not because bookending said you have to.

[marksparkza]

OK, that makes sense. The possibility of an abstract extensible schema sells it for me. :-)

[handrews]

Yeah this could almost certainly all be better. $dynamicRef was an improvement on $recursiveRef but that was a low bar as I got confused by $recursiveRef and I came up with it! But I was really in the weeds while working on it.

For some reason, I really wanted all references to use RFC 3986 URI resolution somehow, and make sense in terms of base URIs and relative reference resolution. I could probably figure out what that reason was, but it might have just been because it's the kind of consistency I like.

At this point, anything that makes it more clear without being less functional would be good. If $dynamicRef could be explained entirely in standard RFC 3986 process terms, I might argue for keeping it, but it's weirder than that. @jdesrosiers if you can simplify it without losing the needed features that would be wonderful.

[jdesrosiers]

I too wanted dynamic references to be URIs, but after after implementing it, explaining it, discussing it, and seeing it abused, I'm convinced that it's just not a good fit. The best solution I could come up with was something like this,

{
  "$dynamicScope": "foo",
  "$ref": "#/$defs/bar"
}

$dynamicScope determines the dynamic scope and uses that as the context URI the $ref will use for resolution instead of the current document's URI. However, resolving a URI against a dynamic context doesn't make sense because you can't make any assumptions about that dynamic context. Therefore, there's no good reason for the dynamic reference to be relative. The only thing that makes sense is to reference an exact dynamic anchor. URI's, unfortunately, aren't a good fit.

[handrews]

OK, finally had time to go over this whole issue again in more detail.

@jdesrosiers your first example is similar to one I'm dealing with for a client right now, so that makes sense to me (although the coupling involved would be substantially more than a single property name).

I had also tried putting in a dummy $defs which is unsatisfying. I like this a bit more in theory:

{
  ...
  "properties": {
    "list": {
      "type": "array",
      "items": {
        "$dynamicAnchor": "list",
        "$dynamicRef": "#list"
      }
    },
    ...
}

As it's very clear that this needs something somewhere to resolve it. But it also needs implementations not to infinitely recurse if they spot this and there's no dynamic anchor of the correct name in scope!

---

However, your links example does not work. Here's your simplified links schema:

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "#meta" }
  }
}

The whole point of this separate schema was to allow people to use the Hyper-Schema links format outside of JSON Hyper-Schema. I am not actually aware of anyone doing that these days, and the decision pre-dates me so I'm fuzzy on what use case might have been offered, but that's the point of that file.

Which means that sometimes, the links schema is the entry point schema, and there is no other resource in the dynamic scope. You must choose a starting target in this case. That doesn't mean there needs to be a bookend, exactly, but you can't just use {"$dynamicRef": "#meta"} either. You have to use a URI that identifies a separate resource, and provides the fragment to look for in that resource.

I remember wrestling with whether it should start at the default schema or hyper-schema. Either way has some sub-optimal effects, but it makes more sense to tie it to hyper-schema. Probably. I think.

Anyway, that's not the point. If you have a $dynamicRef in your entry point schema resource, and no valid target in that resource, then you must use a non-same-document URI reference. There is no way around that.

[jdesrosiers]

@handrews Thank you for taking the time to review this in detail. This is an important issue to me and I haven't been able to get many people involved in this discussion outside of slack.

your links example does not work.

I doesn't work by itself, but it's not supposed to. It's effectively an abstract schema. Using the links schema as an entry point doesn't make sense. You have to provide something with a "meta" dynamic anchor. If you try to use it by itself, you should get an error because "meta" doesn't exist in the dynamic scope. It's no different than a $ref to a schema that doesn't exist. We could give it a default using the dummy $def trick if we want it to have a default. In this case, I think it would be reasonable for the default to be a $ref to the standard 2020-12 meta-schema.

Here's an example of describing an envelope-style media type that includes "headers", "message", and "links" where "links" are borrowed from hyper-schema.

{
  "$id": "https://example.com/custom-media-type",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "headers": {
      "type": "string",
      "additionalProperties": { "type": "string" }
    },
    "message": {},
    "links": { "$ref": "https://json-schema.org/draft/2020-12/links" }
  },
  "$defs": {
    "meta": {
      "$dynamicAnchor": "meta",
      "$ref": "https://json-schema.org/draft/2020-12/schema"
    }
  }
}

The "meta" anchor doesn't even need to point to something that is a JSON Schema dialect. It can be something else entirely and it would still work.

[handrews]

I doesn't work by itself, but it's not supposed to. It's effectively an abstract schema. Using the links schema as an entry point doesn't make sense. You have to provide something with a "meta" dynamic anchor. If you try to use it by itself, you should get an error because "meta" doesn't exist in the dynamic scope.

@jdesrosiers I see your perspective here, but I'm afraid I disagree. It should definitely work. You should not be forbidden from using a schema with a $dynamicRef but no $dynamicAnchor in it as an entry point schema. Set aside the question of whether this particular links schema is useful that way, it's absolutely a reasonable pattern of use.

The dynamic scope starts with the initial target URI. That is part of why the bookending exists. What you have sold me on is the idea that we shouldn't require a resolvable initial target URI. There are clearly also use cases where there is no valid default resolution of the reference. But sometimes there is, and that should work, too.

(and yeah, I feel you on not being able to get a discussion going on these really thorny keywords, and I appreciate your persistence on this)

[handrews]

I guess this is also part of why I wanted it to all be URIs, beyond just the consistency. It was the most reasonable way to resolve the initial target. Of course, there's no reason we can't define analogous behavior with some other mechanism that is more clearly about dynamic resolution.

[jdesrosiers]

@handrews I'm glad we agree on at least some of this. I'm a little confused about what exactly your preference is. I think you said you are ok with the list schema example which has $dynamicRef being used without a $dynamicAnchor, but not ok with links schema example which also has a $dynamicRef without $dynamicAnchor. I see these as exactly the same. Did I misunderstand your position or do you see those examples being different in some way?

[handrews]

@jdesrosiers these are definitely confusing cases! Essentially, the links schema points to a usable $dynamicAnchor, which can be overridden if the same $dynamicAnchor appears earlier in the dynamics scope, while the lists schema does not provide such a "default" dynamic reference target. You have to reference the lists schema from somewhere else (which ends up being earlier in the dynamics scope) which has the right $dynamicAnchor.

This means that links can be used as an entry point (the reference will always resolve), while lists cannot be used as an entry point (the reference will be unresolved without a different entry point supplying a target).

Both cases should work. It's a bit like giving a parameter a default value. It's valid to do that, and it's also valid to require a value to be passed.

[jdesrosiers]

I think we must be talking about different things. Here are the list schema and the links schema together for reference. They use exactly the same pattern. I don't see how one can be ok and the other not.

{
  "$id": "https://json-schema.hyperjump.io/schema/list",
  "properties": {
    "list": {
      "type": "array",
      "items": { "$dynamicRef": "#list" }
    }
  }
}

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "#meta" }
  }
}

---

You should not be forbidden from using a schema with a $dynamicRef but no $dynamicAnchor in it as an entry point schema.

Why? What is lost by not allowing for abstract schemas? I've shown where abstract schemas allow for constraints that can't be expressed otherwise in the list schema example, so I'd need a really good reason to be convinced that they shouldn't be supported. If the opposition has anything to do with concerns about whether or not it can work in a clear, consistent, and predictable way, I can tell you that I've implemented it and know that it works great and is very easy to implement.

Why should a $dynamicRef to nowhere be handled any different than a $ref to nowhere? That seems like a very consistent, predictable and understandable behavior to me.

[handrews]

What is lost by not allowing for abstract schemas?

I stated that you should not be forbidden from using non-abstract schemas (entry point schemas with a $dynamicRef that points to a separate resource with a $dynamicAnchor). That says nothing about not allowing for abstract schemas. Why do you think I am trying to forbid them? If I understand that maybe I can answer your question. Let's just start with this point and then work our way through others one by one.

[jdesrosiers]

Well, then I have no idea what argument you're trying to make.

You should not be forbidden from using a schema with a $dynamicRef but no $dynamicAnchor in it as an entry point schema.

My definition of an abstract schema is a schema that can't be used as an entry point schema. You need to fill in the missing part to make it whole. I see no other way to create an abstract schema than to have a $dynamicRef with no $dynamicAnchor. Therefore, not forbidding any schema to be a valid entry point is not allowing abstract schemas. You must have another definition.

[handrews]

@jdesrosiers I am 100% on board with your usage of abstract schemas. I am also talking about a separate use case: the dynamic, non-abstract schemas that use a $dynamicRef with a reachable $dynamicAnchor. Which does not prevent abstract schemas from existing, nor does it require bookending (although you may end up with a scenario that looks like it, but that's not the same thing as requiring it). They are two uses of the same set of keywords.

Can you help me understand how these two separate use cases conflict? To me, you could:

• allow abstract but not non-abstract
• allow non-abstract but not abstract (this is the case in draft 2020-12, and I agree it is too restrictive)
• allow both
• allow neither (remove $dynamicRef/$dynamicAnchor entirely)

I am trying to demonstrate that we can and should allow both.

[jdesrosiers]

I am also talking about a separate use case: the dynamic, non-abstract schemas that use a $dynamicRef with a reachable $dynamicAnchor.

That sounds like this example.

{
  "$id": "https://json-schema.hyperjump.io/schema/list",
  "properties": {
    "list": {
      "type": "array",
      "items": { "$dynamicRef": "#list" }
    },
  },
  "$defs": {
    "list": { "$dynamicAnchor": "list" }
  }
}

Can you help me understand how these two separate use cases conflict?

There is no conflict. You can define non-abstract schemas just as easily as abstract schemas. So, I guess the next question is why do you think this proposal doesn't allow for non-abstract schemas?

[handrews]

So, I guess the next question is why do you think this proposal doesn't allow for non-abstract schemas?

This is what I am responding to:

I've pointed out that if we remove the bookending requirement, there's never a need for a non-fragment-only URI.

There is definitely a need for non-fragment-only URIs in $dynamicRef because in the non-abstract schema case the target might be in a different resource. Which is exactly what happens with the links schema.

[jdesrosiers]

You're going to have to spell that one out for me with an example. I see no way that the links schema needs a non-fragment-only URI. The example shows how removing the bookending requirement removes that need. Are you saying that something in the links example doesn't work the way I think it will? I assure you that it does. I've implemented it and made absolutely sure that it works.

[handrews]

@jdesrosiers

OK, so if I apply this schema:

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "#meta" }
  }
}

to the instance

{
  "submissionSchema": {
    "type": "integer"
  }
}

what happens?

[jdesrosiers]

what happens?

Error: Referenced schema not found

Same as this schema

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$ref": "/not/a/path/to/a/real/schema" }
  }
}

[handrews]

Right. Because in order for a non-abstract dynamic schema to be usable, you need to support this:

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "hyper-schema#meta" }
  }
}

You can't do that with a fragment-only reference.

[handrews]

And really there's more to this, because the links schema and the hyper-schema meta-schema are mutually recursive, and should both be extensible:

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "$dynamicAnchor": "links-schema",
  "type": "object",
  "properties": {
    "submissionSchema": { "$dynamicRef": "hyper-schema#meta" }
  }
}

{
  "$id": "https://json-schema.org/draft/2020-12/hyper-schema",
  "$dynamicAnchor": "meta",
  "type": "object",
  "properties": {
    "items": {"$dynamicRef": "#meta"},
    "links": {
      "type": "array",
      "items": {"$dynamicRef": "links#links-schema"}
    }
  }
}

You can enter this system from either schema document, so both need to use a non-fragment-only $dynamicRef, because you can't guarantee that either of them is in the dynamic scope for the first pass through the recursion.

This is exactly the situation that motivated dumping $recursiveRef/$recursiveAnchor (which could not handle this) and replacing them with $dynamicRef/$dynamicAnchor.

[jdesrosiers]

Ahhh, now I see what you're talking about. You would do it the same way as you would with the links schema example.

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "properties": {
    "submissionSchema": { "$dynamicRef": "#meta" }
  },
  "$defs": {
    "defaultDialect": {
      "$dynamicAnchor": "#meta",
      "$ref": "hyper-schema"
    }
  }
}

So, you don't need non-fragment-only references, but I can see why you might want it in this case. I think that brings this issue down to preference. Do we keep $dynamicRef simple and focused, or allow for a little syntactic sugar that allows some situations to be expressed more tersely.

[jdesrosiers]

I don't see any reason for the hyper-schema schema to be that complicated. This works,

{
  "$id": "https://json-schema.org/draft/2020-12/links",
  "type": "object",
  "properties": {
    "submissionSchema": { "$dynamicRef": "#meta" }
  }
}

{
  "$id": "https://json-schema.org/draft/2020-12/hyper-schema",
  "$dynamicAnchor": "meta",
  "type": "object",
  "properties": {
    "links": {
      "type": "array",
      "items": { "$ref": "links" }
    }
  }
}

[handrews]

Hyper-schema does not work that way, as both the meta-schema and the links schema must be extensible, and in your most recent formulation, the links schema is not. Mutually recursive extensible things are not uncommon- I'm working with one right now for a client. It's fairly similar- there's a type of extensible thing (A), and there is another extensible thing (B) such that instances of B are used to describe relationships among instances of A. I bring this up to avoid fixating on the specific example of the links schema. It is the mutually recursive extensible pattern that matters.

I will respond to your other comment shortly.

[jdesrosiers]

Ok, I can see why you would want that. I'll try that case and make sure it's working in my implementation.

[jdesrosiers]

This works with my implementation.

{
  "$schema": "https://json-schema.org/draft/future/schema",
  "$id": "https://json-schema.org/draft/2020-12/hyper-schema",
  "$dynamicAnchor": "meta",
  "type": "object",
  "properties": {
    "items": {"$dynamicRef": "#meta"},
    "links": {
      "type": "array",
      "items": {"$dynamicRef": "#links"}
    }
  },
  "$defs": {
    "links": {
      "$dynamicAnchor": "links",
      "$ref": "links"
    }
  }
}

{
  "$schema": "https://json-schema.org/draft/future/schema",
  "$id": "https://json-schema.org/draft/2020-12/links",
  "$dynamicAnchor": "links",
  "type": "object",
  "properties": {
    "submissionSchema": { "$dynamicRef": "#meta" }
  }
}

[jdesrosiers]

This issue was originally about removing the bookending requirement and we did get agreement that that was something we wanted to do. We've yet to come to an agreement about the additional proposed changes that also ended up getting discussed. So, I've created PR #1139 for removing the bookending requirement and Issue #1140 to continue discussion about further changes to dynamic references.