Tightening Datetime, Record Key, and TID validation

[bnewbold]

The atproto specifications define the syntax of datetimes, record keys, and TIDs in Lexicon schemas (both records and HTTP API endpoints).

However, until now the reference implementation has been somewhat lenient about these fields, which has allowed some non-compliant records and client implementations in to the ecosystem.

We will soon start enforcing the syntax more strictly on Bluesky services, at least for API parameters and new records. Our intention is to not immediately break existing records (and URIs, etc), but to stop the creation of such invalid records rapidly.

Our expectation is that some bots and client may start getting validation errors when these changes roll out. Some particular known issues to look for:

• Datetimes missing timezone information, or malformed timezone information. The best option is for datetimes to end in Z, as in 1985-04-12T23:20:50.123Z. A UTC timezone specification of -00:00 is specifically not allowed in atproto. You can read more about datetime syntax at: https://atproto.com/specs/lexicon#datetime
• Record Keys with disallowed characters, like :. There are some notable existing feeds using invalid characters and we intend not to break them, but new records (such as feeds) won't be allowed. You can read more about record key syntax at: https://atproto.com/specs/record-key#record-key-syntax
• TIDs missing zero-padding, or otherwise the wrong length, or containing disallowed characters. You can read more about TIDs at: https://atproto.com/specs/record-key#record-key-type-tid

As a resource for developers, we have a set of test vectors (valid and invalid) as simple text files for all of the atproto identifier type: https://github.com/bluesky-social/atproto/tree/main/interop-test-files/syntax

To deal with the existing invalid records, we may try to pull off a bulk graph update (eg, re-writing records); provide some other migration path; slightly loosen the specification; or just accept some small breakage after a transition period.

Some additional validation tightening which are are not enforcing in this iteration, but intend to eventually:

• Record Key validation against Lexicon. For example, app.bsky.actor.profile records have record key type literal:self, meaning that the record key is expected to always be self. In the future, creating a record in that collection with a record key (rkey) of any other value will be a validation error, the same as having incorrect field types within the record. It would still be possible to disable validation and write a record with such a key, and the repo would otherwise be functional and valid, but the records would valid validation at read time as well.

These are nitty-gritty details, but important to ensure consistent inter-operation between different implementations of the protocol.

[imax9000]

I have one question: but why?

Aside from literal:self, rkey should be just an opaque ID without any structure or meaning for anyone but who generated them.

[imax9000]

records that are created around the same time are placed near each other in your repository

They aren't, MST key contains collection name as a prefix, so records in different collection won't be close to each other.

They'll have lots of overlapping blocks going up to the repo's root, which means proofs can be much more compact.

[citation needed], MST depth should be log(N) regardless of rkey structure.

All in all, please take a step back and stop rationalizing pre-existing status quo and instead consider if imposing much more complex rules on record keys actually makes any sense.

Previously record key was URL path-like string with two components (collection NSID and rkey). Mandating rkey to have a specific length and encoding adds a lots of complexity and custom parsers that haven't been tested by many years of real-world use.

[imax9000]

records that are created around the same time are placed near each other in your repository

Also, this has exactly zero consequences for any practical application, so even thinking about it is a waste of brain cycles and network traffic.

[DavidBuchanan314]

For batched writes, writes with adjacent rkeys will have more shared MST blocks between them, making the overall proof smaller. Even for non-batched writes, the MST implementation is more likely to be hitting warm caches (at all abstraction levels) if consecutive writes have similar rkeys to that of other recent writes.

Consider a thread of posts. Even if each post was made separately, the size of the proof for the whole thread would be smaller than if the rkeys were random.

These are not difficult constraints to enforce, and they make working with atproto data more convenient, efficient, and predictable for everyone.

[imax9000]

You're technically correct (in the first two paragraphs). However I bet that differences in performance/size are negligible.

Are you claiming that this is a solution to an actual problem that the infrastructure had in the past, or that's just premature optimization?

[DavidBuchanan314]

atproto should be able to run at twitter-scale and beyond, and core protocol design decisions made now will be increasingly difficult to change later. It doesn't need much foresight to see that these changes make sense. My only MST-handling code lives inside a single-user PDS, and I haven't benchmarked it, because chasing perf there would be a premature optimisation.

[Zackaryia]

Additionally I believe timestamps should be verifiable to prevent malicious actors and malicious changes. I wrote about this in this discussion. #2166