Parametrized/higher-order/templated schemas

[epoberezkin]

@handrews suggested I submit it as a possible solution to schema extension problem, although I don't think it addresses property extension issue - json-schema-org/json-schema-spec#321 does.

The idea is to support schemas with parameters, e.g.

{
  "$id": "range",
  "$params": {
    "min": { "type": "number"},
    "max": { "type": "number"}
  },
  "type": "integer",
  "minimum": { "$param": "min" },
  "maximum": { "$param": "max" }
}

Then another schema can refer to the first schema using $ref and "pass" parameters:

{
  "type": "object",
  "properties": {
    "foo": { "$ref": "range", "$params": { "min": 0, "max": 9 } },
    "bar": { "$ref": "range", "$params": { "min": 10, "max": 99 } }
  }
}

The second schema can create different constraints by using only one range schema.
The example is trivial and using params in such way is not very useful, but $params become useful with more complex schemas when only a small part of the schema changes based on parameters.

If you compare JSON schema with a function that accepts data and returns boolean, then parametrized schema is a higher-order function - it accepts parameters and returns validating function.

[awwright]

I recall suggesting something similar to this, or at least not as part of the spec, but an idea to generate a schema (server-side) based on query-string parameters.

In any event, I think parameters/arguments in JSON Schema are fairly weak because all the keywords are constraints, and there's little reason to re-use a parameter (without being able to use it in an expression); so if you want to parameterize a keyword, you can just leave it out of the schema entirely, and have the end-user make that constraint using allOf.

This even holds true for "additionalProperties: false" (the biggest problem it seems to me), that's just unpopular because people feel they need to close/cap the collection of properties.

[epoberezkin]

people feel they need to close/cap the collection of properties.

The main reason for that, as I see it, is not capping - in most cases additional properties can be ignored. The real problem that is solved by closing a property collection is typos in property names.

In any case, I don't see how this suggestion can solve this problem - json-schema-org/json-schema-spec#321 solves it.

[epoberezkin]

you can just leave it out of the schema entirely, and have the end-user make that constraint using allOf.

That's not easy when there are several deeply nested values that are set using parameters - at the very least it's very verbose. My use case for this feature is here: ajv-validator/ajv#398 (comment)

[handrews]

This comment was a vote-a-rama comment for considering this as a solution to the re-use-with-additionalProperties problem. But that is not really its main focus, so I've deleted the related commentary. This issue will be revisited on its own merits in the future.

[silkentrance]

Suggestion: rename this to parameterised schema templates, similar to parameterised C++ templates.

This should also work with allOf, oneOf, anyOf...

[handrews]

@silkentrance I added "templated" to the title, as I think all three descriptions fit (assuming @epoberezkin doesn't mind). "Higher-order" is more familiar for JavaScript folks comfortable with functional programming terminology and a very useful mental model for implementations like @epoberezkin's Ajv which compiles schemas to code. But I'd like for more people to see and comment on this so getting folks who jump on because of the "template" concept to comment would be helpful.

[jerstlouis]

A very common use case we have, as part of defining our API in OpenAPI, is to define a list of valid (e.g. enumeration) values which depend on the value of another parameter earlier in the path, e.g.:

/collections/{collectionId}/styles/{styleId}

where the valid values for styleId depend on the value of collectionId

so on one hand we have e.g. a $ref like:

"$ref" : "./api/collections"

to enumerate the possible values for collections, and then we would like to do something like:

"$ref" : "./api/collections-styles/{collectionId}"

Would this proposed approach address such a use case?

[jdesrosiers]

@jerstlouis This proposal is different in that your use case requires values from the instance being validated to fill in the URI Template. The good news for you is that this is exactly what JSON Hyper-Schema does. The bad news is there are no actively maintained JSON Hyper-Schema implementations.

Here's what it would look like.

{
  "type": "object",
  "properties": {
    "collectionId": { "type": "string" }
  },
  "links": [{ "rel": "describedby", "href": "./api/collections-styles/{collectionId}" }]
}

Then ./api/collections-styles/abcd1234 would be something like this.

{
  "type": "object",
  "properties": {
    "styleId": { "enum": ["aaa", "bbb", "ccc"] }
  }
}

[jerstlouis]

Thanks @jdesrosiers that's great to know that something supports this!
In your example, would the href of the describedBy not rather be "./api/collections" ?

Then the "styleId" would be described be a "$ref" or another describedBy with "./api/collections-styles/{collectionId}"?

[jdesrosiers]

The first schema in my example was intended to be ./api/collections. I may have misunderstood part of your example.

[jerstlouis]

@jdesrosiers to clarify, in my examples:
./api/collections right now returns something like:

{
   "type" : "string",
   "enum" : [
      "collection1",
      "collection2",
      "collection3"
   ]
}

and ./api/collection-styles/collection1 would return something like

{
   "type" : "string",
   "enum" : [
      "style234",
      "style235",
      "style236"
   ]
}

while ./api/collection-styles/collection2 would return something like

{
   "type" : "string",
   "enum" : [
      "style454",
      "style455",
      "style456"
   ]
}

That is, the styles that are compatible with a specific collection.

[jdesrosiers]

There's still something missing here. You say you want to

define a list of valid (e.g. enumeration) values which depend on the value of another parameter earlier in the path

JSON Schema doesn't know anything about the URL path a JSON instance was retrieved from. So when you say you have a path like /collections/{collectionId}/styles/{styleId}, I assume that means that the JSON you get from that URL contains a collectionId and a styleId somewhere. That's probably our disconnect. The basic schema for that instance would be something like this ...

{
  "type": "object",
  "properties": {
    "collectionId": { ... },
    "styleId": { ... },
    ... other stuff ...
  }
}

Then you can use $refs and links to fill in those blanks.

{
  "type": "object",
  "properties": {
    "collectionId": { "$ref": "./api/collections" },
    "styleId": { "type": "string" },
    ... other stuff ...
  },
  "links": [{ "rel": "describedby", "href": "./api/collections-styles/{collectionId}" }]
}

[jerstlouis]

@jdesrosiers The use case would be e.g. an OpenAPI definition, so what you get from a /collections/{collectionId}/styles/{styleId} path does not contain a collectionId or styleId, it's a resource that you get by resolving the template with specific {collectionId} and {styleId} values.

Only that the list of valid values by which to replace {styleId} would depend on the value selected for {collectionId} (which OpenAPI & JSON Schema currently does not support).

Hence the essence of the idea would be to support using a parameter id as part of an "$href" to specify the valid values for {styleId} like:

"$ref" : "./api/collections-styles/{collectionId}"

assuming that to resolve that ref you first need to have selected a collectionId (e.g. as you would in a Swagger UI by first selecting a collection from a drop down list, and then the list of valid styles being populated for styleId, which would result in your full path where you could make a request).

Thanks!!

[jdesrosiers]

Ok, I had to look this one up because I'm not well versed with OpenAPI. This is a limitation of OpenAPI. JSON Schema can't make use of data that OpenAPI doesn't pass it. OpenAPI would have to provide a way to validate all path parameters together in order for JSON Schema to be able to validate based on other parameters.

[jerstlouis]

@jdesrosiers Thanks for looking this up!
Correct, to validate some JSON object, you would require e.g. the {collectionId} in my above example.
But the concern is more about how to be able to define such an API, by using JSON Schema snippets as that is what OpenAPI relies on, rather than the validation aspect. The schema could simply be fully resolved by replacing all $ref template IDs (e.g. {collectionId} -> collection1) before the validating step.

The feedback I am getting from the OpenAPI community is that the possibility to implement valid parameters depending on another parameter would rely on this ability to describe this using JSON Schema, e.g. allowing a syntax like:

"$ref" : "./api/collections-styles/{collectionId}"

which in a sense I think is also conceptually a parameterized/templated/higher-order schema, though maybe this goal here is more dynamic in nature as it would be able to query a service directly for valid values, including when those values depend on other values submitted.

[jdesrosiers]

But the concern is more about how to be able to define such an API, by using JSON Schema snippets as that is what OpenAPI relies on, rather than the validation aspect.

The only thing JSON Schema is designed for is validation. When you use a JSON Schema to describe a path parameter in OpenAPI, you are specifying that the value of that path parameter must validate against that schema. So, I don't think we are talking about different things here. We're just looking at it from a different perspective. JsonSchema.validate("collection1", { "enum" : ["collection1", "collection2", "collection3"] }).

You're welcome to clearly define your proposal and submit in as a Github Issue. I really don't see how this can fit into the JSON Schema architecture, but once it's more clearly defined, maybe I'll see something I missed.

I really think that instead of insisting that JSON Schema come up with some way to pass additional context for a validation, OpenAPI needs to include all of the necessary context in what they validate. That means they would need to provide a way to specify a schema that describes all of the path parameters together (probably combined as an object where the keys are the path parameter labels).

[jerstlouis]

@jdesrosiers Well considering the use case of dynamic values, a 'full' schema that details all possible combination could be impractically large (and would need a multi-level dictionaries, where values are again dictionaries of dictionaries etc.).
e.g. you could have /things/{part1}/{part2}/{part3}/{part4} where the possible values depends on part3, part2 and part1, and each of these could have a thousand values.
The nice things with the "$ref" is what we can point to dynamic paths that generate the values dynamically, and we could do so to generate the validation schema for the narrowed down possibilities.
I will file a separate issue for further discussion as the approach seems to differ considerably to the original proposal here, though I believe it is yet another take at "higher order templated schemas". Thanks!