Rename post-split "dependencies" to "schemaDependencies"

[handrews]

Previously, we split the old dependencies keyword in two:

• requiredDependencies is an assertion in the validation vocabulary, and behaves the same as the old string array form of dependencies
• dependencies is an applicator in the core spec's applicator vocabulary, and only supports the old schema form

This makes dependencies incompatible from draft-07 to draft-08: anything using the string form is now incorrect and will not work. This is also confusing for schema authors migrating to draft-08: there is a non-obvious change and they need to change how they think about this keyword.

Instead, let's do a true split, and make the schema case schemaDependencies. This does two things:

1. It makes the behavior clear from the name, in a way that is consistent with requiredDependencies
2. It allows schema authors to continue using dependencies with its pre-draft-08 semantics as an extension keyword

The second point is particularly compelling. It also gives implementors the ability to continue to support the existing dependencies behavior with draft-08 (presumably as an opt-in, or even better with a custom meta-schema using the new vocabulary support allowing schema authors to declare dependencies as an extension- we could even offer a "draft-08-transitional" meta-schema that provides this).

I've been worried about breaking the dependencies keyword in this draft, and I think this is a much better solution for both clarity and migration support.

Thoughts? Paging some folks who weighed in on other dependencies issues: @erayd @markchart @pipobscure

[awwright]

We had properties and additionalProperties before, would the same critique apply to that, even if we didn't have requiredProperties?

[handrews]

@awwright

We had properties and additionalProperties before, would the same critique apply to that, even if we didn't have requiredProperties?

I don't follow. This is about a single keyword being split into two keywords, and the best way to name them while avoiding breaking existing implementations. What does that have to do with properties or additionalProperties? Yes, it's true that properties and required are single words that could be two-word keywords matching the form of other *Properties.

But the point here is not that dependencies is somehow bad because it's one word. The problem is that changing the behavior of dependencies breaks things for people moving from an earlier draft to draft-08. Replacing it with two new keywords allows for compatibility. The symmetry of the resulting keywords is nice but only in the sense of "hey we can allow for compatibility and still make sense", not "we need to make everything a two-word keywords".

[awwright]

You figured it out, I'm questioning if we need keywords to all be two words just for the sake of being the same number of words. I don't see why.

Now if it's a change in behavior, then I see that. Do you have a simple example that demonstrates the change, to make sure I'm understanding it right?

[handrews]

@awwright yeah, that's a reasonable question. The fact that it goes from one word to two is coincidental.

draft-07:

{
    "dependencies": {
        "foo": ["bar", "baz"],
        "biz": {
            "maxProperties": 10
        }
    }
}

We decided in #528 (comment) to split dependencies, keeping the original keyword as the schema form only, with the string array form becoming `requiredDependencies. This was done in PR #573, resulting in:

{
    "requiredDependencies": {
        "foo": ["bar", "baz"]
    },
    "dependencies": {
        "biz": {
            "maxProperties": 10
        }
    }
}

The problem here is that now dependencies has shrunk in functionality. It looks like the same keyword, but it's really not. That' sub-par UX for people migrating from older drafts. And as it turns out, a number of implementations do not switch on draft number, really, but have logic that handles draft changes on a per-keyword basis. So keeping dependencies as the same keyword with different behavior complicates that. And even when you do switch globally on draft version, it's annoying. And there's no sensible way to implement some sort of transitional/compatibility mode.

These are things that are not compelling on their own, but they add up and slow adoption.

So what I would like for draft-08 to look like is:

{
    "requiredDependencies": {
        "foo": ["bar", "baz"]
    },
    "schemaDependencies": {
        "biz": {
            "maxProperties": 10
        }
    }
}

There is no chance of mistaking either of those keywords for the old dual-function dependencies. And it would allow continuing to support the old dependencies syntax simultaneously if people want to do that.

So, to summarize:

• We've already done the split, which is the change in behavior
• We haven't locked in the new names (Move "dependencies" to applicability group #528 (comment) even notes that we'll quite likely rename them further as we think about it more)
• Splitting the old keyword into two new keywords, instead of splitting off a new keyword and changing the old keyword, is a much better migration story.

[handrews]

Since I can't get anyone to comment further on this despite reaching out in several forums, I'm moving to a PR.

[erayd]

I missed emails on this sorry - will comment on it tonight.

[handrews]

Thanks, @erayd. See also PR #614, feel free to comment wherever seems most useful.

Having a PR doesn't make this inevitable.

[erayd]

@handrews
My preference would be to keep things as dependencies and requiredDependencies. I don't really want requiredDependencies to exist at all (although I can understand why people might desire it) - the entire purpose of having requiredDependencies at all is just a giant UX kludge to support the following:

{
    "dependencies": {
        "foo": {"required": ["bar", "baz"]}
    }
}

---

To me, this change really looks like a behavioral change in dependencies that simplifies its definition (from schema | array to schema only), but keeps the same behavior. That should not IMO result in a keyword change.

The problem here is that now dependencies has shrunk in functionality. It looks like the same keyword, but it's really not.

How is it not the same? It looks the same to me, the only change is removing a syntactic shortcut.

It also gives implementors the ability to continue to support the existing dependencies behavior with draft-08 (presumably as an opt-in, or even better with a custom meta-schema using the new vocabulary support allowing schema authors to declare dependencies as an extension- we could even offer a "draft-08-transitional" meta-schema that provides this).

Why does it need renaming for this though? There's nothing stopping implementations from continuing support for pre-draft-08 syntax; it's just a simple type switch (i.e. if the value is an array, either warn / error or provide deprecated support).

I agree that UX should be a priority, however I think that the eventual UX of the final standard is more important than the UX of migrating from one draft to another. dependencies is a much more obvious keyword than schemaDependencies, and I'm very reluctant to discard it just to handle an edge-case when migrating between two pre-release drafts of the standard.

As far as moving between drafts goes, making $schema mandatory would go a long way towards improving this, as it would require schema authors to declare the version they are using, rather than possibly requiring the implementation to guess or make assumptions if $schema is absent.

As it turns out, a number of implementations do not switch on draft number, really, but have logic that handles draft changes on a per-keyword basis. So keeping dependencies as the same keyword with different behavior complicates that. And even when you do switch globally on draft version, it's annoying. And there's no sensible way to implement some sort of transitional/compatibility mode.

Could you expand on this a bit? I really don't see how this is complicated from an implementation perspective. The backwards-compatibility logic is easy - if definitions is an array, and using deprecated features is allowed, then rewrite it to {"required": $array}. Done. And that could be either on-demand or in preprocessing; there's nothing tying that logic into the actual keyword implementation.

[handrews]

I don't really want requiredDependencies to exist at all

We already lost that argument a long time ago, no point in revisiting.

There's nothing stopping implementations from continuing support for pre-draft-08 syntax; it's just a simple type switch (i.e. if the value is an array, either warn / error or provide deprecated support).

In everything I've ever implemented (an assortment of bits and pieces, most not public), dependencies is always the ugliest mess in the code, so I disagree with your dismissal of this concern.

dependencies is a much more obvious keyword than schemaDependencies

I strongly dispute this. In my experience, dependencies has always been very confusing for people. And even more importantly, I would be willing to bet a lot that most preferences for dependencies can be explained by preferring what is familiar over what is new.

I really don't see how this is complicated from an implementation perspective.

It's complicated compared to other keywords. The original dependencies is a horrible mess from my point of view because it conflates assertions and applicators, and I want to remove any vestige of that.

Also, implementing exclusiveMinimum and exclusiveMaximum across multiple drafts is really annoying.

Your objections seem to boil down to not liking change, but the whole point of being in pre-release drafts is to facilitate change. I try to avoid it as much as possible (goodness knows there are other things I'd personally like to tweak), but when we have decided to make a change (as we have here), I want to make the change clear and obvious, and facilitate migrations as much as possible.

What I'm looking for here is an affirmative argument for keeping it as dependencies: Something compelling that is far better with dependencies and far worse with schemaDependencies. I'm not hearing that. If the discussion is personal preference, or some people thinking that the code is easy and others thinking that it's hard, I'm going to go with the easier option.

[erayd]

We already lost that argument a long time ago, no point in revisiting.

Fair call :-).

In everything I've ever implemented (an assortment of bits and pieces, most not public), dependencies is always the ugliest mess in the code, so I disagree with your dismissal of this concern.

Any chance you could expand on this? The logic is relatively straightforward, so I'm not sure why it should result in a messy implementation. Is there something obvious I'm missing here?

The ugliest bits of implementation I've had to do with have mostly been:

• exclusiveMinimum (because of the bool -> int change),
• exclusiveMaximum (because of the bool -> int change)
• $ref (because of non-standard reference target locations in compiled schema, and impact from $id)
• additionalProperties (because of the interaction with properties and patternProperties)
• items (because of the schema | schema[] dual type nature)
• oneOf (because of error bubbling)
• required (because of the change from boolean to array),
• pattern (because of the ECMA-262 regex requirement)
• format (because of the complexity necessary to implement some of the options).

In my experience, dependencies has always been very confusing for people.

Do you know what aspect of it has been confusing? It seems intuitive to me (i.e. if $property, then apply $schema or if $property then require $otherProperties), but perhaps I'm missing something?

It's complicated compared to other keywords.

It's complicated compared to some keywords, sure - but way less complicated than others. What is complicated about the following?

for each ($dependencies as $propertyName, $definition) {
    if ($definition is array) {
        if (pre-draft-08 || allow deprecated) {
            $definition = {"required": $definition}
        } else {
            throw error
        }
    }
    if ($instance has $propertyName) {
        validate ($instance against $definition)
    }
}

The original dependencies is a horrible mess from my point of view because it conflates assertions and applicators, and I want to remove any vestige of that.

I agree completely. That is what removing the array type addressed - it's now a simple keyword that applies a schema, or not, based on whether a property is present.

Your objections seem to boil down to not liking change, but the whole point of being in pre-release drafts is to facilitate change.

Perhaps I haven't been clear enough - I will try to rephrase. I'm not objecting to change, so much as objecting to change without a good reason. I haven't seen a compelling enough reason to do so here.

I try to avoid it as much as possible (goodness knows there are other things I'd personally like to tweak), but when we have decided to make a change (as we have here), I want to make the change clear and obvious, and facilitate migrations as much as possible.

Agreed.

What I'm looking for here is an affirmative argument for keeping it as dependencies: Something compelling that is far better with dependencies and far worse with schemaDependencies. I'm not hearing that. If the discussion is personal preference, or some people thinking that the code is easy and others thinking that it's hard, I'm going to go with the easier option.

1. schemaDependencies makes me think "stuff that my schema depends on" (like external refs to definitions or something). dependencies makes me think "list of dependencies". Neither are perfectly clear, but the latter feels like better UX, and has the advantage of being existing vocabulary - users already know that it means "list of dependencies".
2. The behavior of dependencies has not changed - it still applies a list of constraints based on whether properties are present. Changing the name when the behavior has not changed seems unwise, and more likely to confuse users.
3. The default position should be dependencies because it is the status quo, and there is user inertia associated with that. I have nothing against change, but it should be for a reason that is compelling enough to be worth confusing everyone who already knows how dependencies works. Making this change forces them to go and find a new keyword to do the same job, so we shouldn't do that lightly.
4. Changing the name in this case sets a troubling precedent - namely that whenever a keyword's syntax is changed in a minor way, even though the behavior doesn't change, then the keyword name should also be changed in order to facilitate backwards-compatibility.
5. Changing the name in order to facilitate backwards-compatibility feels like an attempt to preempt what implementations may want to do. IMO the spec should not be forcing that kind of decision on people - let implementors handle this in whatever way they prefer. As long as the spec does not force excessive complexity or resource overhead, then we should avoid interfering.
6. The decision regarding whether to rename from a UX standpoint should be based on the UX of the final standard, not the UX of shifting between drafts. While the latter is a consideration, it's short-term, whereas the final standard will have a much longer term impact.

The problem is that changing the behavior of dependencies breaks things for people moving from an earlier draft to draft-08. Replacing it with two new keywords allows for compatibility.

I would like to address this specifically. Things breaking between drafts should be expected - it's a draft standard under development. Schema authors migrating from an earlier draft to a later one should expect to make changes to their schema. Implementation maintainers should expect their implementation to require new logic. Compatibility is important. However compatibility, where implementations desire it (and according to the spec they should) can be easily maintained without a name-change in this case.

Going forward, attempting to use an array definition here will simply result in a type error. It's not a silent failure, it's a screamingly obvious your-schema-is-invalid error. If the change were behavioral in nature (e.g. if we kept the array form, but it now meant something else) then the potential for a silent change in the validation outcome would make the rename far more compelling - but as it is, keeping the name as-is does not cause any silent issues for users at all.

[handrews]

schemaDependencies makes me think "stuff that my schema depends on" (like external refs to definitions or something).

This is a fair point, and I am certainly open to other names. That are not dependencies.

All the rest of this appears to be personal preference, based on differing notions of what constitutes good UX. And I can't help but feel that if I'd just done the names differently in the first PR that this would never have even come up, and we're stuck in this argument because I slipped up there.

---

I am not going to further discuss the questions of what is messy or not, because it is subjective and we clearly have different views.

Changing the name when the behavior has not changed seems unwise, and more likely to confuse users.

The behavior has changed. It no longer accepts a list of strings. If you do not believe that that constitutes a change, then I don't see any productive direction for this conversation. All of this is predicated on the fact that dependencies does not have the same behavior as it did in draft-07.

However compatibility, where implementations desire it (and according to the spec they should) can be easily maintained without a name-change in this case.

No, it cannot. The string array behavior is gone, and an implementation that allows it would be in violation of the spec. However, if the split keywords are two new keywords, then dependencies, from the point of view of draft-08, is an extension keyword, and can be handled in whatever way is desired.

let implementors handle this in whatever way they prefer

That is what I am doing- getting the new keywords out of the old, so that implementations can do whatever they want with the old.

Changing the name in this case sets a troubling precedent

All of these things are on a case-by-case basis, and validation and core keywords have barely changed at all in the last three specs. There is no requirement to do anything. I think it makes sense in this case.

The default position should be dependencies because it is the status quo

The status quo is already changed, this makes no sense at all to me.

Going forward, attempting to use an array definition here will simply result in a type error. It's not a silent failure, it's a screamingly obvious your-schema-is-invalid error.

We have different notions of what constitutes good UX here.

[handrews]

@erayd I don't know what to do with this. It's basically down to a difference of opinion. I look at this as fixing a bug in the originally agree-upon change, you don't. I honestly thought this would be totally uncontroversial, but I guess nothing ever is. Everything on this project is a giant argument, but at the end of the day I have to write the PRs and publish something. Do I decide in my favor because I'm writing the PRs? Do I avoid deciding in my favor because I write the PRs? Which gives everyone a veto when the differences aren't big enough to be obvious? I have no idea anymore.

[erayd]

All the rest of this appears to be personal preference, based on differing notions of what constitutes good UX... We have different notions of what constitutes good UX here.

I think this may be the fundamentally the cause of us disagreeing on this.

The behavior has changed. It no longer accepts a list of strings. If you do not believe that that constitutes a change... The status quo is already changed, this makes no sense at all to me.

I believe it constitutes a syntactical change, not a behavioral one. Whereas you consider a syntax change to be a kind of behavioral change.

I don't know what to do with this. It's basically down to a difference of opinion. I look at this as fixing a bug in the originally agree-upon change, you don't. I honestly thought this would be totally uncontroversial, but I guess nothing ever is. Everything on this project is a giant argument, but at the end of the day I have to write the PRs and publish something. Do I decide in my favor because I'm writing the PRs? Do I avoid deciding in my favor because I write the PRs? Which gives everyone a veto when the differences aren't big enough to be obvious? I have non idea anymore.

As far as I'm concerned, if you feel that an executive decision is required to move things along, feel free to make that decision in your favour, and overrule my input. You carry much more of the load of this project than I do or ever have, and seem likely to continue to do so in the future. I may not like your decision, or agree with it, but I don't want to stall progress simply because I disagree with you :-).

I've made my opinion known, but it is an opinion, and I am only one voice. You have proven to me on many occasions that you listen, and have good reasons for your decisions. As such, I'm quite happy to let you decide what the path forward should be.

[handrews]

Thanks, @erayd. I'm poking one or two other people for a sanity check, as I'm always suspicious of myself when I want to overrule someone without a compelling technical reason. If I can't get that then I'll move ahead as-is.

Regarding syntax vs behavior: I think entirely in terms of assertion/annotation/applicator classifications. dependencies was a dual assertion/applicator, and now (if not renamed), it's purely an applicator. The syntax is irrelevant. I'm not really trying to sell you on this so much as to explain my rationale. I'm not sure anyone else has really bought the keyword classification stuff, but it kind of underpins nearly everything I'm doing in draft-08, so I base everything on that conceptual framework.

[erayd]

I'm poking one or two other people for a sanity check, as I'm always suspicious of myself when I want to overrule someone without a compelling technical reason. If I can't get that then I'll move ahead as-is.

That sounds like a good plan 👍

Regarding syntax vs behavior: I think entirely in terms of assertion/annotation/applicator classifications. dependencies was a dual assertion/applicator, and now (if not renamed), it's purely an applicator. The syntax is irrelevant. I'm not really trying to sell you on this so much as to explain my rationale. I'm not sure anyone else has really bought the keyword classification stuff, but it kind of underpins nearly everything I'm doing in draft-08 so I base everything on that conceptual framework.

I think I get what you mean here - am I correct in my understanding that you're saying that it's less about behavior, and more that this change has fundamentally changed what class of keyword this is? And that because the keyword class has changed, that is inherently a major change, even though the cause of it was syntactical?

[handrews]

I think I get what you mean here - am I correct in my understanding that you're saying that it's less about behavior, and more that this change has fundamentally changed what class of keyword this is? And that because the keyword class has changed, that is inherently a major change, even though the cause of it was syntactical?

Yes, exactly. To me, class == behavior. You're looking at it as a generalized "if property X is present, then take action Y". From that perspective, it's the same, because you don't care that much about what Y is. I think. I care far more about the nature of Y. "apply this subschema to the current location" and "assert that this property is present" are completely different things to me. The fact that there's an "if property X is present" involved in both is irrelevant. Or perhaps not irrelevant, but just coincidental.

[erayd]

You're looking at it as a generalized "if property X is present, then take action Y". From that perspective, it's the same, because you don't care that much about what Y is. I think.

Yep, that's accurate - I'm seeing it more from the perspective of "what logical steps are required to implement this", and I wasn't thinking about the class distinction at all.

Are you able to expand a bit more on why you feel the class distinction is so important (feel free to decline, or do so via email, if you think that's off-topic for this thread)? In a more general sense I mean, not so much specifically about this keyword - more around why it matters so much [to this project], when it's fundamentally an abstract concept. I think this is something that's fuelled many of the other debates on this project, so it would be useful to better understand your perspective on it.

[erayd]

This is a fair point, and I am certainly open to other names. That are not dependencies.

What about dependentSchemas? It's fairly self-evident as "a list of schemas which depend on something else" (in this case the presence of properties on the instance).

[handrews]

@erayd I've been trying to figure out a way to get that point across for a while, with little success. It's a diversion, but I might as well give it a shot here- if necessary I can delete it and file it elsewhere, but it's not like a ton of people have been commenting on this one :-)

It has to do with transitioning JSON Schema from a primarily validation-oriented tool (with other things kind of as afterthoughts) to a modular and extensible system, with consistent keyword mechanisms and behaviors.

In particular, generative use cases (UI generation, code generation, and documentation generation) are very well-established in the wild. More so than hyper-schema, really. Hyper-schema's primary use over the last several years has been for documentation, rather than runtime link generation. For most of these cases, what was needed to improve support for them was not more validation keywords, but other sorts of information (e.g. "this allOf is expressing inheritance" vs "this allOf is just for schema organization and should not impact code generation directly", or "display fields in this order", or "use widget X for this field", none of which affect the validation outcome at all).

The appropriate thing to do seems to be to acknowledge that those are important use cases for JSON Schema as a system, and make them first-class citizens alongside validation. But while validation assertions are well-established, annotations have always been very underspecified. This has particularly been a problem with default, but it extends to other things as well. It especially has never been clear what to do with multiple annotations that apply to the same location- does one title overwrite the other? What about description? Do you concatenate them? There is no clear answer- the ideal behavior is highly situational. At one point I proposed a whole suite of default-ish keywords for different behaviors, which fortunately @awwright shot down (this is why I get nervous about doing what I want- some of my ideas are actually terrible).

The best approach was to be more clear about what information JSON Schema implementations are expected to present to applications, and be clear that application should define how they make use of that information. See PR #610 and issues #530 and #396 for more details on this.

The other part of this is applicators (a term that I don't really like, but haven't come up with anything better- and no, they don't all have to start with "a" although that does amuse me). Way back in draft-04, there was this awkwardly worded section about calculating children schemas, and a really complicated description of object validation. Array validation was nearly as complicated. @awwright cleaned that up tremendously in draft-05, but it got me thinking about why those keywords were treated like that in draft-04.

Fast-forward a ways, and I'm talking with a maintainer of a UI generation library, who points out that while he often needs properties or items he doesn't necessarily need validation assertions (it was phrased differently, but that's what it meant). And it occurred to me that you could easily write a hyper-schema without any assertions if you were just using it to put links on, and didn't care about validation at all (why you would do this is another question entirely). So this was the motivation for moving the applicator keywords over to core. They behave differently. Once I have time to really hone the language, I think they will be much easier to explain. And with this, Hyper-Schema (and potentially other things) no longer depends on validation. You can, and probably will, use it with validation, but the spec itself no longer has that dependency. This seemed like a really good confirmation that the classification system was useful.

Finally, the whole unevaluatedProperties thing (#556) required really sorting out annotations and applicators and how they could interact without just making some really messy special cases that would be far more complex than additionalProperties. And also (#602) figuring out some further distinctions like in-place vs child applicators, which answer the question "how do I figure out what other keywords need to be checked before I can use unevaluatedProperties?

I don't know if that helps at all- I figured I'd try narrating how and why I got here rather than trying to lay out the theory. In general, it's helpful for both answering questions of "where does this go? Is this within the current scope or a new thing? etc." and for spotting design smells like dependencies. And also format which has weird dual optional behavior that confuses nearly everyone (still working on sorting that out- it won't involve a keyword change AFAIK, though).

[handrews]

@erayd it's funny you should mention dependentSchemas as dependentRequired was considered in #528 for the split. The rationale for using requiredDependencies was (lightly edited for clarity out of its original context):

If the string list value is the list of "dependencies" of the instance property, than "requiredDependencies" is better (also closer to the current name that implies the same). If it's the opposite (the instance property is a "dependency" of each property in the string list), then "dependentRequired" is better.

I had preferred dependentRequired but went with the other option mostly to avoid further argument. Perhaps we should rethink that. I would be perfectly happy with dependentSchemas and dependentRequired.

[erayd]

@handrews

I would be perfectly happy with dependentSchemas and dependentRequired.

Assuming that we are proceeding with moving away from dependencies, then I like the idea of naming them similarly. dependentSchemas and dependentRequired sounds good to me. There's a nice symmetry to it :-).

Regarding your long reply on classes - that's really helpful. I'd like to re-read some of the other discussions in that light, so will probably take a couple of days to digest it all, but it's giving me a much better handle on where you're coming from. Initial tentative first impressions are that (shockingly!) I may agree with you ;-). Will respond to that in a bit more detail once I've finished going over stuff.

[handrews]

OK, I'm going to sleep on it but I like the dependent* names. I think it actually works, too. The instance property MUST be present in order for the string array or schema to be used. So the string array and schema are dependent on the presence of the property.

I'll note that I even commented in #528 that we could tweak the names later, as I was not all that confident of them, but wanted to move forward for a variety of reasons that aren't relevant anymore.

[handrews]

Having slept on it, I like dependent* a lot better, so I'm going to take down the existing PR and rework it accordingly.

[Relequestual]

Having read all relevant comments, I'm in favour of splitting into two new keywords, prefixed with dependent. Sounds and feels reasonable to me.

[handrews]

This got merged back in June but I forgot about the issue. Closing now.