Lexicon: validating unknown $types in open unions

[snarfed]

Hi all! I'm trying to parse the lexicon validation language around open unions. https://atproto.com/specs/lexicon#field-type-definitions says:

By default unions are "open", meaning that future revisions of the schema could add more types to the list of refs (though can not remove types). This means that implementations should be permissive when validating, in case they do not have the most recent version of the Lexicon.

OK. Lexicon changes have to be backward compatible over time, so validating a known lexicon permissively seems like the default behavior. However, what should validation do when an open union field value has an unknown $type? I assume ignore it, ie allow anything that's valid within the data model, since we don't yet have runtime lexicon resolution?

https://bsky.app/profile/moji.blue and https://bsky.app/profile/poll.blue are examples here, afaict they both store non-app.bsky objects in union fields in app.bsky records. I don't have example records handy but I can grab some if we need them.

[bnewbold]

Great question! And timely.

First some background: a little while back we put out a small change on PDS validation behavior, which hasn't been reflected in written specs yet:

• Revise repo write validation for unknown Lexicons #2748
• https://bsky.app/profile/bnewbold.net/post/3l6fyxsw7za2o
• lexicon: document PDS validation behavior atproto-website#353

tl;dr: when creating a record on PDS, by default it is optimistic: will validate if a schema is available, but allow if not available. "mandatory" validation can be enabled, and "skip validation" can be enabled. The PDS will reply with a flag indicating if validation happened or not.

We don't currently have a strong conception of "recent" / "latest" Lexicon schema: as you mention they need to be backwards compatible. There is a bit of a de-facto difference between "published" and "pre-published", but at least for records, they need to start compatible.

The way this intersects with unions is that we are leaning towards open union validation working like the "default" state: validate if any version of the schema is available, but "allow" if not. This would probably be true for open unions even "mandatory" validation was selected at record creation time in the PDS request.

There might be an additional concept of "mandatory recursive validation" which would apply to open unions as well; and a concept of "if you don't know the schema, go out and resolve it" (we are working on Lexicion resolution right now). These flags might just be SDK-level debugging flags though, not sure if those will ever be PDS query params.

Big picture, we think we probably don't want PDS instances to have to do "live" Lexicion resolution in the middle of processing record creation, even with layers of caching/indexing to help. We might add (authenticated?) endpoints to ask a PDS to resolve lexicons on demand, which would update any local cache (and result in the default behavior updating to do validation for those lexicions), so clients/users can drive this process. This ties in to how auth scopes are going to work, where the PDS needs to implement permissions defined in a lexicon schema.

[snarfed]

Thanks! This all makes sense. I'd seen the validate flag on putRecord, the three-way option makes sense, and also that optimistic applies to open unions and unknown types. And I like not requiring or blocking on lexicon resolution in critical write paths. Seems like that idea is always catnip to ontology-wonk standards body types - witness COM/CORBA, XML, JSON-LD, etc 🤪 - but infamously they tend not to be working engineers who've built and run systems at real scale and understand the tradeoffs.

...er, sorry for the rant. I'm done now. 😎

[Tamschi]

It's a matter of use-cases. The simplified record scheme only works because views are global on ATProto.

For something like ActivityPub, JSON-LD or similar is necessary to create an extensibly collision-free protocol (and arguably caching known, named contexts is much more efficient than always retransmitting them there, since you can efficiently cache their transforms in memory).

I agree on XML though, that one's just horrible to actually comply with to the highest degree.