Using annotations as a mechanism to describe inter-keyword communication

[gregsdennis]

Related:

• Clean-up of stable features and removal of unstable features json-schema-spec#1444 (action items from stable features discussion)
• Proposal to use annotations as a descriptive mechanism for keyword comms
• specify if, then, and else together using annotations as interaction mechanism json-schema-spec#1451

I may have jumped the gun in using annotations as a descriptive mechanism for keyword comms without properly seeking opinions.

While I don't want annotations to be prescriptive in this way, I think they're a handy way to describe how keywords interact.

Moreover, I think consistency is important, so (ideally) I'd either like for all dependent keywords to have their behavior described using annotations, or for none to use annotations.

The example I think makes the case for using annotations is unevaluated*.

I'll just take unevaluatedProperties as the example, but unevaluatedItems has the same problem.

unevaluatedProperties will evaluate properties not covered by properties, patternProperties, or additionalProperties found as either a sibling keyword or in passing subschemas of sibling keywords which the keywords themselves produce a true validation result. It even works through static ($ref) and dynamic ($dynamicRef) references.

Statically (without an instance) determining which properties to evaluate is generally impossible.

{
  "if": {
    "properties": {
      "foo": true
    },
    "required": ["foo"]
  },
  "then": {
    "properties": {
      "bar": true
    },
    "required": ["bar"]
  },
  "else": {
    "properties": {
      "baz": true
    },
    "required": ["baz"]
  },
  "unevaluatedProperties": false
}

If foo is present, then bar must also be present and nothing else; otherwise baz must be present and nothing else. This depends on the instance, so static analysis can't determine which properties unevaluatedProperties applies to.

Without annotations, evaluating this means that an implementation would have to evaluate if/then/else, taking note of which properties were evaluated in a passing subschema, then provide that list of properties to unevaluatedProperties.

But that's exactly what the annotations of properties, patternProperties, and additionalProperties are already doing. These keywords produce an annotation that is a list of the properties they evaluate. Plus, if their subschemas ultimately fail, the annotations are dropped, so we only get the ones from passing subschemas. Not using annotations is just internalizing the exact same mechanism, except that it requires implementors to "re-invent" that logic. Since the implementation has to track this data anyway, why not describe it using an existing mechanism that does exactly what the implementation needs?

[jdesrosiers]

The unevaluated* keywords are definitely dependent on annotation behavior. They are unique in that respect. I have no problem with using annotations to specify keyword behavior. However, I don't think they should be exposed to the user in the output. That's not their purpose. I'd be ok with saying they "MAY" be exposed in output if implementations find it difficult to filter them out, but it shouldn't be expected and definitely shouldn't be required.

[gregsdennis]

However, I don't think they should be exposed to the user in the output. That's not their purpose. I'd be ok with saying they "MAY" be exposed in output if implementations find it difficult to filter them out, but it shouldn't be expected and definitely shouldn't be required.

I'd prefer we refrain from veering off-topic. This is to discuss whether the specification should use annotations to describe keyword interaction.

[jdesrosiers]

I disagree that it's off-topic. The main issue for me with using annotations to describe keyword interactions is that that choice doesn't leak into output requirements.

[Relequestual]

The main issue for me with using annotations to describe keyword interactions is that that choice doesn't leak into output requirements.

Having annotations in the output format allow secondary tooling to understand the logic workflow. This is actually really critical information. Sure, some implementations could provide some other format to explain exactly what they did and why, but that won't enable generic debugging tools that work with multiple implementations (that support the output format).

In summary, we absolutley SHOULD have these sorts of annotations in the output format.

If you want to catagorise them into "logical decision annotations" and "data information annotations", then that's probably OK, and maybe we could work on that. I vaguely recall having a similar discussion some time ago along these lines.

[jdesrosiers]

I see your point. There's value in having that information for debugging purposes. However, having that work consistently would mean that we would have to require that implementations produce those annotations. I believe that we have agreed at this point that we don't want the spec to prescribe implementation details, specifically the case of using annotations as inter-keyword communication. Those two goals are not consistent and I think requiring the use of annotations for inter-keyword communication is more harmful than a theoretical debugger that might be able to make use of those of those annotations. Some of the consequences of using an annotations-based architecture are discussed in https://github.com/orgs/json-schema-org/discussions/204.

[Relequestual]

Going in this direction stems back to Henry's work on the "JSON Schema Framework", evaluation of behaviours and suggestions for improvements.

Taking json-schema-org/json-schema-spec#1451 as an example, aka if/then/else.
Henry's evaluation went like this:

if/then/else as written is a custom interaction. You write code to handle those specific keywords togehter.

Henry broke down annotations into three classes, one of which was "explanatory". The suggested solution was so that

• if's assertion result is used as an Explanatory aNnotation (EN)
• then and else become "adJacent consUmers" (JU) of if's annotation result

This creates a behaviour based understanding of how the keywords interact, and means you are not writing custom interaction code, and can implement the keywords independantly from one another (although you still need to know the order of keyword resolution).

Why does this matter? A few reasons.

One of the most challening aspects of updating an implementation (so I'm told) is when interactions between keywords change, or new interaction requirements are added.

By thinking about requirements of an implementation in terms of behaviours, once you implement the behaviours, you just need to call the behaviour related functions when a keyword needs them. (A real nice separation of concern.)

Additionally, someone writing a vocabulary/extension could want to access any of the behaviours. It would make it easier on them to be able to use existing behaviour and not reinvent the wheel. It also makes sure that additional vocabularies have access to the data required to make use of different types of behaviours.

[gregsdennis]

there's just one example of that

unevaluated* is the only one that has a dynamic dependency, sure, but it's not the only keyword with dependencies. It's just that the others (e.g. additionalProperties) only depend on the contents of their dependencies, not the evaluation result. Thus those keywords can statically inspect their dependencies, which can be done easily in isolation. Implementing additionalProperties with annotations transforms that static dependency into a dynamic dependency.

[gregsdennis]

Regarding if/then/else and json-schema-org/json-schema-spec#1451, specifically, the spec currently lays out these requirements:

• if keyword is a no-op since its subschema's validation result doesn't affect the schema's validation result; it can be skipped entirely on its own.
• then keyword has to know the validation result of if's subschema in order to determine whether it should evaluate its own subschema.
• else keyword has to know the validation result of if's subschema in order to determine whether it should evaluate its own subschema.

This layout promotes the duplication of work. It promotes parallelization. Any argument that using annotations is influencing implementations is invalid because the current text already influences implementations to prefer a parallelizable, but work-duplicating architecture.

---

unevaluated* is the only one that has a dynamic dependency

I was wrong here. then and else also have a dynamic dependency in that they rely on the evaluation result of if.

[Relequestual]

I'm holding out on some research before I fully reply, however...

By thinking about requirements of an implementation in terms of behaviours, once you implement the behaviours, you just need to call the behaviour related functions when a keyword needs them.

I don't think it works this way in real life. I'd be interested to know if there are implementations that actually implementation keywords using reusable behavior related functions. I've not seen that to be useful. All of my keywords are implemented independently and I have no cases were I feel like I'm duplicating logic. My understanding is that the goal of the classification system was more to identify the existing behaviors so we can use them as a canary to warn us that a new keyword being proposed may not be implementable in all implementations.

Having been involved in the discussion as it was evolving, I'm pretty confident that was indeed one of the primary reasons: making implementations easier to maintain by reducing the complexity of interactions and code required for those interactions.

[jdesrosiers]

This means that "properties", "patternProperties", "additionalProperties", and all in-place applicators MUST be evaluated before this keyword can be evaluated.

Here's a different way to phrase this that I believe is based on observable behaviors rather than internal mechanisms.

"The set of properties that are considered "unevaluated" MUST take into account all keywords in the schema that result in keywords being considered "evaluated" regardless of where unevalutedProperties appears in the schema."

Maybe someone can wordsmith that to make it a little more readable, but I think that captures the intent without assuming how keywords are evaluated by an implementation or assuming a mechanism by which the set of "evaluated" properties is constructed.

Of course other text is necessary to describe what makes a property "unevaluated". I think it's also possible to describe that in a way that doesn't use annotations, but there's probably little value in trying to do that. I'd probably describe it using annotations and make a note that implementations can do whatever they need to as long as the behavior is the same.

[Relequestual]

I said I'd come back after some research, so here I am.

It does look like we've reached a way forward and something we agree on, but I wanted to just shine a light on some further context.

Why do you have to do the work twice? The short answer is that a dependency exists, and that dependency necessitates an order of evaluation.

The long answer is that unevaluatedProperties still needs to know what properties to evaluate. To do that, it needs to run other keywords first. There are two strategies for this:

* Run `unevaluatedProperties` in isolation, meaning that you have to first run its dependencies within that isolated context.

* Run `unevaluatedProperties` in series after all of its dependencies have run in a global context.

In both cases, you're running unevaluatedProperties after its dependencies.

The theory still holds: you have to wait for a keyword's dependencies before you can run the keyword. - @gregsdennis

Yup, this is accurate. However there has been some cross talk, saying that this is not correct.

I think what's happening here is different assumptions were made (including myself), and that has made it harder to understand each others positions. So in reality, I think we are all right/correct! Let me explain.

I chatted with Julian a fair amount yesterday, digging into his implementation and how it handled unevaluatedProperties.

It handles it in isolation from other keywords, meaning it can be done in parallel, but the function itself also does all the validation for itself. This means that validation for keywords can be logically run independently, but work is duplicated. The duplicated work is thrown away.

I made the assumption that duplication of work in this way wasn't something that would be published. I was wrong. (Maybe Greg also made this assumption?) Julian made that trade-off knowing the impilcations. Y'all are smart people! Working code that passes tests is better than "perfect" code that isn't released. Besides, it was unclear if the feature might change in some way, so making costly architectural changes for something that might also change, is an unwise investment.

Jason's approach is similar in terms of logic, but the architecture is built up to be much more dynamic, and closer to what I imagined. There's a cost to maintaining something a little more complex though. Jason also made trade-offs which he understood, and it is built that way for a reason.

In some chats with Julian, I proposed that keywords could wait for others to finish (using Promises in javascript for example), removing the duplication. Julian pointed out a number of challenges involving this, specifically the need to build and support a DAG for execution. I've built interconntected async await using systems that are trying to resolve and have dependencies, and we didn't need to know ahead of time a resolution order explicitly, but Julian pointed out that it would likely be more performant if you did.

I can't think of a legitimate analogy to make a summary. What I'm seeing is, Julian's and Jason's implementation use the first of the two strategies Greg suggested were viable. But they also allow for indiscriminate parallelization later.

Hopefully this is more useful than not, and convinces us, and others, that we were actually closer than we assumed.

[ssilverman]

you simply do work twice

Why do you have to do the work twice? The short answer is that a dependency exists, and that dependency necessitates an order of evaluation.

The long answer is that unevaluatedProperties still needs to know what properties to evaluate. To do that, it needs to run other keywords first. There are two strategies for this:

• Run unevaluatedProperties in isolation, meaning that you have to first run its dependencies within that isolated context.
• Run unevaluatedProperties in series after all of its dependencies have run in a global context.

In both cases, you're running unevaluatedProperties after its dependencies.

The theory still holds: you have to wait for a keyword's dependencies before you can run the keyword.

I think it’s useful to use the phrase “as if” here. Implementation order only needs to behave as if execution was in a certain order, i.e. the user only needs to see the results as if there’s an order. Some language specs use this to allow compiler optimizations.

[gregsdennis]

I see where you're going here and I like the idea.

But there is distinctly an order of evaluation in this case. You can't run unevaluated* without first having run its dependencies because you need the dependency results to know what's "unevaluated."

[gregsdennis]

What I mean is that we can say "as if," but that doesn't mean the hard dependency doesn't actually exist. I think I'd prefer that we explicitly state the dependency.

[Relequestual]

I'm looking at Julian's implementation as an example. It looks like it does not first run dependencies for unevaluatedProperties, but I don't see how that can be true AND it passes the tests. I've asked for an explanation. I didn't want to clog up the other thread with such request and resulting discussion, so I'll report back =]

[gregsdennis]

I had a thought that stemmed from my analyses presented in this thread above. (I think I just talked myself out of using annotations as the mechanism, but cases like if/then/else should be rare.)

Currently there are multiple kinds of dependencies:

1. A keyword depends only on the presence/contents of another. This would include things like items depending on prefixItems and contains depending on min/maxContains.
2. A keyword depends on the evaluation result of another. This is how then and else work: they depend on the evaluation result of if's subschema.
3. A keyword depends on the annotation result of another. This is how unevaluated* work.

I'm calling (1) a "static" dependency since they can generally be assessed without an instance.

I'm calling (2) and (3) "dynamic" dependencies because their assessment requires an instance.

---

(Just thinking out loud a bit for better communication, for those who want to try to parse this.)

I was just about to propose that we require keywords which are dynamic dependencies where the dependency is on the evaluation result (e.g. if is a dynamic dependency of then) be required to emit that evaluation result as an annotation. However, that's putting a requirement on the dependency (if) in order to satisfy the dependent (then). That can't always work because someone could potentially define a keyword that depends on the validation result of minimum which, within our specs, has no dependents, so we're not going to make it emit an annotation. Likewise a third-party vocab can't require minimum to emit that annotation, so their keyword can't do what it needs to.

However, defining a third-party keyword that depends on the validation of minimum isn't going to work anyway because if minimum fails validation, then the local schema fails validation, making the third-party keyword moot.

That means that a keyword that has an evaluation-result dependency (e.g. if) MUST NOT affect the evaluation result of the local schema, and the only reason to have such a keyword is to serve as a dependency for another keyword (e.g. then/else). Therefore it's safe to assume that the dependent keyword and its dependency, for evaluation-result dynamic dependencies, will always be defined together.

So the only reason to have an evaluation-result dependency is within a (closed?) multi-keyword system that is designed together. (We wouldn't define if by itself because it does nothing.) In such a system, does it make sense to require the dependent keywords to emit their validation result as an annotation? Probably not.

I could see a case for independently defining an informative keyword that behaves like if but produces an annotation of whether its subschema failed. Supposedly that could be useful to the consuming application. But at that point it's producing an annotation anyway, which if a third-party keyword depends on that keyword, it becomes an annotation-result dependency, not necessarily evaluation-result.

---

I think the solution is for json-schema-org/json-schema-spec#1445 to allow all of

• static dependencies
• evaluation-result dependencies *
• annotation-result dependencies

These are the spec-defined mechanisms that keywords may use to interact.

* Evaluation-result dependencies and dependents MUST be defined together because the dependency keyword MUST NOT affect local schema validation, and it doesn't make sense to define a keyword that doesn't affect local validation result (and doesn't produce an annotation) unless you have another keyword that depends on that behavior.

---

Second question: does it make sense to restrict third-party vocabularies from inventing their own interaction mechanisms?

[jdesrosiers]

This is great! The concept of static vs dynamic dependencies associates to meaningful architectural properties. This is a great way to break things down.

does it make sense to restrict third-party vocabularies from inventing their own interaction mechanisms?

Is it even possible have something other than those? That seems to cover all the bases. Assuming I'm wrong, I would say no. At most we can say that if third-parties invent something other than what is defined that they run the risk that it might not be possible to implement their extension in all implementations.

[Relequestual]

Sounds reasonable.
If if didn't effect the evaluation result, these tests would need to be modified, right? Because currently, the annotations of subschema value of if bubbles up if the assertion is in the positive.

[gregsdennis]

if's subschema's validation result doesn't factor into the validation result of if's local schema.

[Relequestual]

Agreed. if does not add an assertion into the overall validation result.

The subschema's validation result does impact unevaluatedProperties though.

Therefore, it is not accurate when you say...

if keyword is a no-op since its subschema's validation result doesn't affect the schema's validation result; it can be skipped entirely on its own.

I'm not sure that SHOULD be the case. I've always found it a little akward. I'm in favour of changing that if others would be, but it would be a change in actual validation result, and a change in tests.

(I actually think that having a property only defined in if that you expect to be able to trust is not a great idea/best practice.)

[jdesrosiers]

does it make sense to restrict third-party vocabularies from inventing their own interaction mechanisms?

Is it even possible have something other than those?

I was having a serious lack of imagination when I said that. Keywords could be defined to have dependencies on configuration (format-annotation) or an external data source such as a database or API.

[chanced]

The challenge I have with annotations driving logic of sibling keywords is order of operations. Each of my keywords produces an output node that is processed by the parent schema / keyword. The keywords do not have access to sibling output at the time of evaluation.

I realize that this isn't going to happen, but man would it be easier if things were grouped hierarchically.

• $props could be an object with unevaluated, pattern, required, min, max, additional, etc.
• $cond could be an object with if, else, then
• $items could contain all of the item related keywords
• ... and so on.

If structured this way, you'd have keywords which are composed of all co-dependent sub-keywords. For example, an implementation could have their $props keyword accumulate all evaluated properties from all of the positive-testing keywords and pass those to unevaluated, when applicable, before finalizing it's own evaluation.

The need to maintain state or re-evaluate for sake of unevaluated* would dissipate.The only state an implementation would need throughout an evaluation is the locations of $dynamicAnchors.

This behavior can be emulated today, with the current structure, using transient nodes but it is confusing.

[jdesrosiers]

It would certainly be easier to implement if all keywords involved in a dependency where grouped, but it would make things more verbose. For things like the properties and items keywords, I don't think that trade-off is worth it. However, that's only because those are all static dependencies. A static dependency is where you can get all the information you need without evaluating the JSON instance. You don't have to worry about order of operations with static dependencies. For example, additionalProperties can look at properties and get the property names it declares without evaluating properties against the instance. Instead of needing to worry about order of operations, you can just grab what you need when you need it form the schema.

Dynamic keywords like if/then/else whose interactions are dependent on an instance are a different story. Putting those under one keyword would definitely make life easier for implementers. It would be no harder than evaluating anyOf. No special inter-keword communication necessary.

I'm not sure grouping actually solves the unevaluated* problem. This is the worst of dynamic keywords because it's effectively in a dependency group with every applicator.

[chanced]

I was completely wrong about this.

I hadn't gotten far enough along with my implementation consider the implications of unevaluated* when paired with the likes of if / then / else, refs, and so on.

I realize this topic is closed but I just wanted to mention it incase anyone stumbles upon it in the future.

[karenetheridge]

I don't know why this discussion is closed. It doesn't say who closed it. But anyway...

Having annotations in the output format allow secondary tooling to understand the logic workflow. This is actually really critical information. Sure, some implementations could provide some other format to explain exactly what they did and why, but that won't enable generic debugging tools that work with multiple implementations (that support the output format).

In summary, we absolutley SHOULD have these sorts of annotations in the output format.

Why? Why are annotations necessary at all? I've never found a use for them. They're just an annoying "not-an-error" in result output.

The fact that unevaluatedProperties uses annotations is simply because the keywords it depends on (properties etc) happen to produce the annotations that contain precisely the information that unevaluatedProperties needs to make its decision. But why does properties generate this annotation at all? Who has ever used it for anything? unevaluatedProperties could get its information another way if it needed to (something internal, not exposed to users, that for these keywords happens to behave just like annotations). After all, we maintain other internal state information that we don't seem to question, like the dynamic scope used for $dynamicRef and $dynamicAnchor to function properly.

I'm questioning the conflation of two things which are really separate: annotations as exposed to the user in the result, and annotations as an internal thing for passing state information between keywords and subschemas.

[gregsdennis]

I closed it because of #491 (comment)