Behavior negotiation: Balancing author intent and runtime flexibility

[handrews]

NOTE: This approach did not resonate with folks so I am considering this discussion closed. I will try a different approach to the idea at some point.

A primary goal of vocabularies is to increase the confidence of schema authors that their schemas will be processed as expected, or not at all. Of course, someone using the schema can edit the meta-schema or (depending on the implementation) override it to get around those requirements, but doing so most likely involves observing a failure to run and making a choice to circumvent it. That's totally fine, and sometimes there are good reasons for doing that sort of thing.

We also have some spec-mandated as well as commonly-present runtime configuration options that are outside of the control of the spec author, and in various discussions folks have proposed more of these. Let's figure out some architectural principles for this. Let's not discuss mechanisms, unless you think that something is outright impossible to do.

My starting point:

• Schema authors MUST be able to specify a range of acceptable behaviors, as completely as possible.
• Any mechanism for specifying such things MUST NOT be burdensome to those not using it.
• Implementations MAY offer configuration options as long as they respect the specified range.
• Implementations MAY offer the ability to go outside the range, but doing so MUST involve some additional option or configuration that makes it explicitly clear that the schema author's intent is being circumvented, and MUST NOT be enabled by default or as an automatic consequence of some less clear option.

I think that these are pretty solid principles, but I'm definitely open to counter-proposals as I came up with them fairly quickly.

Here are some examples to help make this more concrete — I'd like to avoid trying to solve these specific problems in this discussion, but we can refer to them while discussing if the principles make sense.

How format does and does not comply with the above principles

With 2020-12, we almost got format to comply with these principles. Schema authors can select format-annotation or format-assertion.

While even format-assertion's specification acknowledges that validation behavior might still vary, it is a stronger requirement than how people implemented format in the past, and it's clearly acknowledged in the specification. A schema author who requires format-validation knows that they will get a substantial amount of validation for every standard format, and accepts that the spec allows some variation.

Therefore, any variation in format-assertion behavior that is within the spec's allowances is acceptable runtime variation.

However, format-annotation does not guarantee that format will only be treated as an annotation. It's possible to turn on the old-style best-effort format validation regardless of how the schema author set up format-annotation in $vocabulary. If a schema author wants to outright forbid any validation attempts, there is no way to do that.

So while we can hand-wave and say "yeah, format-annotation includes that, so it's still within the schema author's specified range of behavior", doing so defeats the purpose of format-annotation as the predictable option.

However, if running best-effort validation on top of format-annotation required an explicit "circumvent schema author intent" option (either separately or in the name or at least documentation of the option itself), then that would be fine.

Current problem: annotation support

Right now, a schema author cannot require annotation support. This matters if the validation concerns of the schema are pretty trivial, and the main work is being done in complex annotations. Chances are, the person running the schema would figure this out, but it's an example of someplace where more control would be better.

Annotation collection is still a somewhat experimental feature, and not universally supported. If you really depend on it, it would be better to get an error (or at least warning?) so that someone not deeply versed in JSON Schema could quickly figure out that they need a different implementation, or need to lobby for annotation support.

This likely generalizes for other use cases involving not-yet-stable features as discussed in #234.

[jdesrosiers]

I don't have an opinion on this yet, but here's a perspective to consider.

The context in which a schema is used can be significant. Assume a schema is used in a web application to validate form data. The schema is used on the front-end to give user feedback before they submit, but it's also used on the back-end to enforce that the submitted data is valid.

Let's assume that the implementation used on the front-end doesn't support format as assertions (neither as vocab nor config), but the implementation on the back-end does. It's not critical that the front-end implementation catches format errors as long as the back-end implementation catches them. It's not the best user experience, but that's not the point.

If the schema is written to require format-assertion, the front-end implementation that doesn't support this vocab can't work at all providing an even worse user experience than a best-effort validation. If the schema is written to make format-assertion optional then a back-end implementation that doesn't support format-assertion could get a false positive validation result because format-assertion behavior should be required on the back-end.

This is all mostly not a problem in a closed system where the schema author is also the consumer of the schema, but that's not always the case. The schema author doesn't always know the context in which the schema will be used, so putting behavior requirements solely in the control of schema authors can be limiting.

[handrews]

This proposal allows the schema author to accept a range of appropriate behavior.

@jdesrosiers in your example, where the schema is being used in a client-server system, the author can be aware of the differing possibilities. They can do any number of things to manage it:

• allow format to behave either way in that schema and document that anyone who uses it with an implementation that does not validate format is expected to perform the additional validation some other way
• write separate client and server schemas that differ only in their requirements around format behavior
• require that format be treated as an assertion because they specifically want any client that can't do that to fail as quickly and clearly as possible.

Even if the schema author chooses the last option, someone who absolutely wants to use the schema anyway can easily get around it by running with a modified meta-schema (or whatever the control mechanism is). There is no reason to prevent a schema author from requiring fail-fast behavior. Fail-fast is a common architectural principle in environments where variable behavior can have catastrophic consequences.

[handrews]

That 3rd bullet point should have said treated as an assertion, not treated as an annotation 🤦 I've fixed it now, my apologies.

[jdesrosiers]

where the schema is being used in a client-server system, the author can be aware of the differing possibilities

Yes, it wasn't the best example. The point is that a schema author isn't always aware of the different contexts the schema might be used in.

[handrews]

The point is that a schema author isn't always aware of the different contexts the schema might be used in.

Why is that a problem? When I write a library in code, I have certain use cases in mind. I have no idea what people might do with it. But I am not obligated to design the library to accommodate everything that anyone might want to attempt.

For example, if a JavaScript library requires a new feature isn't supported in a browser, and that can't (or just hasn't been) polyfilled (is that still a thing? it used to be a thing), it should test for that. If it's not available, the code should fail with a clear indication that it can't run due to lack of support. Yes, a library written for maximum adoption would have some sort of fallback, but not everything needs that level of contingency planning.

[handrews]

(JavaScript and browsers may not be an ideal example given the tendency of web code to muddle on through and try to do as much as it can, but hopefully you get the point)

[gregsdennis]

I'm of the belief that the specification defines a behavior, and through configuration away from the default mode, implementations can do what they want.

Basically a compliant implementation:

• MUST implement the spec-defined behavior as the default behavior
• MAY provide an option to behave differently
• MUST document any alternate behaviors and the configurations needed to achieve them

If a feature must have multiple modes defined in the spec, then the spec must also declare one of those modes as the default. We also need to be aware that if the spec defines multiple modes for a feature, it puts a burden on implementors to support all of those modes and a configurability mechanism.

[handrews]

@gregsdennis thanks for commenting. I don't really disagree with what you've said, I'm not entirely clear on how it fits with my proposal. This proposal is about allowing the schema author and the person/code running the evaluation to negotiate the behavior.

I'm going to write out an example, which is also intended to help readers who may not have as much context or background, so I'll be pedantic about it and assume readers might not know HTTP well.

HTTP content negotiation as an analogue

In HTTP content negotiation the client expresses preferences and the server answers within that set of preferences. For our purposes, the directionality ends up reversed: the client is analogous to the schema author (setting the bounds of allowable behavior), while the server is analogous to the JSON Schema implementation (answering within those bounds).

Let's assume that our server documents the following defaults, which are the only things guaranteed to work across all resources (these are the MUSTs):

• Resources are represented in XML
• Responses are encoded with br (the Brotli algorithm)

It documents a variety of other possible formats and encodings, but offers no guarantees that any particular resource supports them (they are MAYs)

Note that the identity encoding (doing nothing) is always supported- it's a MUST at the HTTP level, not the API level.

Now let's assume that a client prefers to work with JSON, preferably with a JSON Schema associated. It also (for... Reasons) doesn't have a library that can decode br encoding, but it can handle gzip.

This asks for the resource http://example.com/resource as application/schema-instace+json if possible, with application/json the next-preferred option, and application/xml as a last resort (if none of these are available the result will be a 406 Not Acceptable. It asks for gzip encoding if possible, and for no encoding otherwise.

GET /resource HTTP/1.1
Host: example.com
Accept: application/schema-instance+json;q=1, application/json;q=0.9, application/xml;q=0
Accept-Encoding: gzip;q=1, identity;q=0

Since the client has allowed for both a format (XML) and an encoding (identity) that are guaranteed by MUST requirements, it can be confident of getting a 200 OK response unless something goes wrong on the server side.

Here the server responds, indicating that it did not have application/schema-instance+json, but was able to provide application/json, and was able to gzip it.

200 OK
Content-Location: /resource
Content-Type: application/json
Content-Encoding: gzip

JSON Schema and format control

Currently, the schema author can require the format-assertion vocabulary, or can use (required or optional) the format-annotation vocabulary. If the author wants format-assertion or refusal to process, we know how to do that.

But with the format-annotation vocabulary, the implementation can be configured to attempt some vaguely-defined level of format validation anyway. There is no way for the schema author to control that, and there is no way to tell whether it was done or not when you get the result. What if we want to forbid any attempt to validate?

Perhaps in the license agreement for the API, there is language indicating that clients MUST perform their own format validation with an additional library that is provided as part of the customer SDK, which understands the basic and hierarchical output formats. This actually could be a legal requirement protecting the API provider from liability if, for example, this is a financial transaction and the client fails to validate the transaction timestamps (or something).

Let's pretend that in addition to $vocabulary, we have a $requiredConfig (this is not a proposal for such a keyword, just an example off the top of my head):

{
    "$vocabulary": {
        ...,
        "https://json-schema.org/future/format-annotation": true
    },
    "$requiredConfig": {
        "annotationCollection": true,
        "formatValidation": false,
        "output": ["basic", "hierarchical"]
    },
    ...
}

This indicates that the JSON Schema implementation MUST be able to collect annotations, MUST be able to produce either basic or hierarchical output (not just flag), and MUST NOT attempt to validate format. If we didn't care whether format validation was attempted or not, we could have left that out (again, this is an example, I'm making it up as I type, please don't focus on whether this $requiredConfig would actually be a good solution- I haven't thought it through at all, and I'm not gung-ho to add more keywords).

A JSON Schema implementation will refuse to process a schema referencing this meta-schema if any of the following are true:

• It can't collect annotations at all
• It can't produce either basic or hierarchical output (either because it doesn't support them at all, or because it was not configured for them and cannot convert the result afterwards - if the result can be converted, then the implementation would process the schema even if the pre-configuration did not select either of those output formats)
• It was configured to attempt validation in a way that did not allow the implementation to disregard that setting

This ensures that the result of the validation can be used in accordance with the licensing requirements. In this case, the output format itself will make clear which of "basic" or "hierarchical" was used, but for other options (for example, if we said we didn't care whether format validation was attempted), including those settings in the output would probably be good (I had not thought of this until just now).

The proposal as written still allows the implementation to offer an explicit override. So if someone really wants to run this through an implementation that doesn't support annotation collection or the standard outputs, and wants to force it to attempt some amount of format validation, they can still do that.

But they have to be explicit about it, rather like how rm won't automatically recursively force-remove an entire directory tree. You have to tell it to do that specifically.

Key points

• This is about allowing negotiation rather than putting all power with one side or the other
• Implicit or accidental overriding of author intent is prevented
• No functionality is lost, because users can always explicitly override the author allowances

The main use case here is not format (I hope we can just drop the weird half-assed option), but experimental features per the proposed SDLC. A schema author may want to enforce an experimental feature's use. Depending on the feature, that may not otherwise be apparent from reading the schema. For example, there's no way to tell from a schema whether the annotations present will actually need to be used as annotations or not. They may just be there as documentation.

[gregsdennis]

I don't really disagree with what you've said, I'm not entirely clear on how it fits with my proposal.

Your title is "config options." Over the past few weeks, we've been discussing implementations having options that enable behaviors outside of the spec (or even defined by it, like for format). Consumers of an implementation could fiddle with the knobs to get an alternate behavior. Ensuring these options have the right defaults (*cough* ajv) has been the core of our recent discussions.

I think you're talking about something else, and the context switch isn't apparent from your original post.

[handrews]

@gregsdennis I'm confused here. This issue is about what I said in the opening comment. If the words "config options" mean a different topic to you, I'm sorry for the confusion — titles are hard. Please work with what I'm actually proposing here. If you're not sure what I'm proposing, please ask questions.

[gregsdennis]

There seems to be a confusion between what I read this to be and what it is. I will continue trying to wrap my head around the topic.

[handrews]

@gregsdennis I have changed "config options" to "behavior negotiation", which hopefully at least reduces the feeling that this discussion ought to be about something else. Config options are part of behavior negotiation, but not all of it. And this is not about the specifics of any particular config option, or what config options should look like. It's more about how they (and any other controls like the vocabulary system) can or should be used in general. Which we have never defined.

[handrews]

After some discussion, @gregsdennis and I worked out that the confusion was over what is being configured. Greg thought we were talking about non-specification, while this issue is actually talking specifically about configurations within the specification. Due to that misunderstanding, this comment will be marked off-topic and we'll continue in other comment threads.

[handrews]

NOTE: This approach did not resonate with folks so I am considering this discussion closed. I will try a different approach to the idea at some point.