JSON Hyper-Schema Needs Schema Version Parity

[cjaccino]

In release 2020-12, JSON Schema released new draft schemas without updating Hyper-Schema. As a result, Hyper-Schema's most recent drafts are associated with 2019-09. Given the differences between 2019-09 and 2020-12, as well as differences in implementations, the lack of schema parity for Hyper-Schema discourages its adoption. I propose that even without feature changes to Hyper-Schema, its meta-schemas and related materials should continue to be updated and released as part of the JSON Schema cohort of specifications.

Hyper-Schema is a clean, graceful, portable way to provide hypermedia support for JSON objects. Unlike other hypermedia styles, it avoids changes to the objects themselves. Though it has not been widely adopted thus far, growing interest in JSON Schema is likely to increase attention to its ability to support hypermedia.

Please consider bringing Hyper-Schema back into parity with JSON Schema.

If you agree, please 👍 this request.

Thanks for all the great work on JSON Schema. You are making a difference!

[jdesrosiers]

I think this is a reasonable request. It's normal practice for IETF drafts* to be republished without changed after they have expired to indicate that they are still active. We don't have the bandwidth to maintain Hyper-Schema right now, but the effort to simply republish with with updated schemas would be trivial. I'd volunteer to do the work.

• Although we aren't affiliated with IETF in any way, we do roughly follow their practices.

[jdesrosiers]

Another approach to consider is that we don't even necessarily need to publish a new draft. All we need is to provide a new dialect meta-schema that uses the most recent version of validation and hyper-schema. Both are designed as modular vocabularies for exactly this kind of thing.

For 2020-12, we would need to provide a 2020-12 version of the 2019-09 hyper-schema vocabulary schema because of issues with $recursiveRef and $recrusiveRef being gone in 2020-12, but that's not something that would need to be done for every release. At that point it would be trivial to construct a dialect schema with the 2020-12 validation vocabularies and the 2019-09 hyper-schema vocabulary. Technically, anyone could do this and still be confident about compatibility with anyone else who builds their own dialect schema. Us doing it would be a courtesy for the community.

[cjaccino]

@jdesrosiers I personally am fine without an updated IETF draft. Updated meta-schema would be enough to avoid breaking implementations and keep the door open to an upsurge of interest. With OpenAPI's recent adoption of 2020-12, there will be a lot of new attention on JSON Schema, and it seems like an inopportune time for Hyper-Schema to fall out of view.

[jdesrosiers]

We talked about this and decided to go with what I proposed and publish a dialect that uses 2020-12 validation with 2019-09 hyper-schema. But, we have a choice to make. It turns out that the links.json schema uses $recursiveRef wrong. The only allowed value for a $recursiveRef is "#", but the links.json schema uses an absolute URI. In fact, $recursiveRef isn't even capable of expressing the relationship that was intended. However, $dynamicRef is capable. So, we have a choice to make.

1. Leave hyper-schema 2019-09 meta-schemas broken.
2. Update the hyper-schema 2019-09 meta-schemas to use 2020-12. It's a little strange to have a meta-schema in a future draft, but it's the only way to get the intended functionality of allowing hyper-schema to be used with custom dialects.
3. Fix the hyper-schema 2019-09 meta-schemas and pin them to 2019-09 validation.

I prefer option 2 because it makes maintaining things in the future easier. The downside is the strangeness of using a meta-schema for 2019-09 that uses a dialect that didn't even exist when 2019-09 was released. Since what we have now in unusable anyway, it's unlikely we would be stepping on anyone's toes with this change. But, I'd be ok with any of the other options as well if anyone feels strongly about it.

[karenetheridge]

How difficult would it be to do a full new version of hyperschema that used $dynamicRef and the rest of the draft2020-12 dialect?

[jdesrosiers]

Not difficult at all. All of these options are easy. It's just a matter of deciding how or if we want to fix the 2019-09 schemas since what was originally intended is not possible.

[Relequestual]

This would be a good usecase for using the $comment keyword to explain what the actual heck we did and why 😅

[Relequestual]

@jdesrosiers do you feel we need a separate Discussion relating to broken 2019-09 hyper schema meta-schemas? If so, please feel free to create one and link to it form here.

[jdesrosiers]

@Relequestual If it looks like there's going to be multiple opposing strong opinions that need to be discussed, I'll create a discussion. But, I don't think that's likely.

[cjaccino]

Thanks for doing this, Jason!
I'm not overly opinionated on the how, but I'd hope the approach was as helpful as possible for tooling. For example, mixing 2019-09 and 2020-12 might be a problem for AJV, which, as far as I know, can't do both in a single instance.

I also wonder if it would make sense to manage hyper-schema independently from the rest of JSON Schema. Decoupling might avoid the problem of hyper-schema being a burden to JSON Schema and the risk to hyper-schema as a going concern. Hyper-schema could simply trail JSON Schema's release schedule and develop independently. Not sure I see all the implications, but thought I'd put that out there.

Thanks again!

[jdesrosiers]

You have a good point about AJV's limitations. That's something to keep in mind for the future, but in this case, we will need to provide a 2020-12 version of the 2019-09 Hyper-Schema vocabulary anyway because $recursiveRef and $dynamicRef don't mix. The thing we have to decide is what to do about the 2019-09 dialect which is currently broken.

I also wonder if it would make sense to manage hyper-schema independently from the rest of JSON Schema.

Effectively, it is independent. It's its own specification and absolutely can move at it's own pace. The only way that the Hyper-Schema spec is a burden to Core and Validation specs is that it takes time away from Core and Validation work. We're just stretched too thin.

[jdesrosiers]

I don't like the idea of just leaving 2019-09/hyper-schema broken and since AJV would likely choke if it was written in 2020-12, I think that leaves option 3 as the way to go.

3. Fix the hyper-schema 2019-09 meta-schemas and pin them to 2019-09 validation.

[karenetheridge]

since AJV would likely choke if it was written in 2020-12

Isn't that AJV's problem? (meant half-humorously). I think the fact that hyperschema's use of $recursiveRef is not according to the spec is a compelling reason to bring it up to draft2020-12 and use $dynamicRef.

[jdesrosiers]

Isn't that AJV's problem?

Yes, it certainly is. I'm not super concerned about AJV specifically especially because there are alternatives that can do the job. I was thinking along the lines of, if AJV has this limitation, then there are probably more out there that do as well and some of those implementations might not have good alternatives. It creates a higher barrier of entry even if it might have the positive effect of forcing implementations to improve.

Then again, we don't want to encourage people to use 2019-09/schema at this point anyway, so maybe making it harder to use it with hyper-schema is a feature not a bug.

Also, no matter what we choose, it's not going to work with AJV until they fix their $dynamicRef implementation anyway.

[jdesrosiers]

I'll leave the two PRs for adding and fixing hyper-schema meta-schemas up for one more week to give people a chance to comment, approve, object, or whatever. Then barring any objections, I'll merge them then.

[jdesrosiers]

The schema fixes and additionals have been merged and I've opened PR json-schema-org/json-schema-org.github.io#419 to make them available on the website.

[cjaccino]

@jdesrosiers Jason, thanks again for your work on this!

[jdesrosiers]

Schemas are now available on the website.

[karenetheridge]

Should https://json-schema.org/draft/2020-12/output/hyper-schema exist?

[karenetheridge]

Also, share/draft2019-09/links.json has "$schema": "https://json-schema.org/draft/2019-09" which is invalid -- this should either be ".../hyper-schema" or ".../schema".

[jdesrosiers]

Should https://json-schema.org/draft/2020-12/output/hyper-schema exist?

No, https://json-schema.org/draft/2020-12/hyper-schema should exist. It does.

share/draft2019-09/links.json has "$schema": "https://json-schema.org/draft/2019-09" which is invalid -- this should either be ".../hyper-schema" or ".../schema".

🤦‍♂️ Yeah, it should be ".../schema".

[karenetheridge]

https://json-schema.org/draft/2019-09/output/hyper-schema exists.

$; wget https://json-schema.org/draft/2020-12/output/hyper-schema
--2021-11-08 13:46:23--  https://json-schema.org/draft/2020-12/output/hyper-schema
Loaded CA certificate '/opt/local/etc/openssl/cert.pem'
Resolving json-schema.org (json-schema.org)... 104.21.8.16, 172.67.130.91
Connecting to json-schema.org (json-schema.org)|104.21.8.16|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2021-11-08 13:46:23 ERROR 404: Not Found.

[jdesrosiers]

That's a 404 response, right?. That's the response I would expect for any URL to something that doesn't exist. Am I missing what you're trying to tell me?

[karenetheridge]

https://json-schema.org/draft/2019-09/output/hyper-schema exists.
https://json-schema.org/draft/2020-12/output/hyper-schema does not exist.
This seems like an oversight.

[jdesrosiers]

Oh, I didn't realize "output/hyper-schema" was a real thing. I guess I should add that too.