Issues with long strings in atproto records

[bnewbold]

There is no specific maximum size for a record, or a string in a record, in atproto. Implementations are likely to have some limit, whether intentional or not: many CBOR libraries have default object size limits and other precautions against DDoS attacks, and often HTTP servers (or frameworks) have a body limit somewhere.

Giving a specific limit in the specification itself, either for records or strings inside records, seems arbitrary and inflexible (could cause interoperability problems later). On the other hand, leaving it ambiguous is already starting to cause problems.

To make things more concrete, there are valid records in the network today with strings over 19 KByte in the external embed description string field. The CBOR library used in the golang implementation rejects these long strings, preventing inter-operation with those records.

This is a discussion thread to acknowledge the issue and discuss it publicly.

[snarfed]

Agreed that ideally ATProto itself should have few or none of these limits, but they're probably appropriate in many app-specific lexicons, right?

Otherwise, agreed, leaving it to HTTP servers and CBOR libs reminds me of network path MTU discovery. Is it 1400? 1500? 1480? 1280?

URL max length may be a counterexample. afaik the RFC(s) don't specify it, and browser disagree, but that doesn't seem to cause too many problems in practice. How long can URLs be? "Long enough."

[bnewbold]

a fun one would be including an exponential inflation rate in the spec itself, with the current max size being a function of the current year.

[DavidBuchanan314]

I think URL lengths are a good example of Postel's Law in action - and unlike MTUs, there's little incentive to push up against the limits unless you're doing something particularly weird

[bnewbold]

FWIW, AT URIs and components (DID, NSID, rkey) do end up in database primary keys, and URLs, and that motivated slapping on some (generous) size limits in the specs.

[DavidBuchanan314]

Last time I checked, there's a 100KB JSON request size limit for bsky.social, which acts as an indirect limit to how big your CBOR objects can be (as an implementation detail, and not based on the spec). In an ideal world, I think it would make the most sense to have an explicit maximum for overall CBOR size, but no limit on the size of individual objects within it - based on my guess that that logic is easiest to implement, usually (since it's well-defined, and you can perform a single check up-front, without needing a "smart" CBOR parser capable of enforcing per-object limits as it parses). The maximum could either be declared in the spec, or declared by each service (a PDS could advertise the maximum record size it will intentionally accept/emit, and an AppView could advertise the maximum it will accept, etc. etc.)

Out of curiosity, how much effort would it take to raise whatever limit is causing the golang impl to reject 19KB strings, until it can accept 100KB strings? (since that's the unofficial maximum you'll ever see from bsky.social, at least)

[DavidBuchanan314]

As an aside, I enjoy having sensible maximum limits when writing software. I've been aiming to copy that 100KB limit in my own PDS, and it's informed my design decisions (for example, it's small enough that I don't need to care about trying to stream cbor decoding, and it also limits DoS vectors as long as I can handle ~50K nested arrays or objects - which I can!)

[bnewbold]

We can bump the limit in the CBOR library, should not be a real problem, just friction.

Max size for any field (string, bytes, etc) being the max size of entire CBOR makes sense. I agree that specifying a max CBOR limit probably makes the most sense. My intuition/guess would be a couple MByte, not 100 KByte, but could be wrong. I think we're being conservative with JSON limit, and focused on microblogging app; other applications might gain a lot from a size bump without having to go all the way to blobs. Eg, GPS tracks (GPX), long-form text (especially non-Latin script) like fan fic. Blobs always there as an escape hatch.

[DavidBuchanan314]

A maximum object nesting depth might also be worth defining (I believe some of the records in my own bsky.social repo caused Issues during the AppView migration, due to differing implementation limits).

[bnewbold]

This feels to me like something that folks will either just do on their own regardless of spec (if they use a CBOR library that supports it), or ignore even if it is in the spec. Would we expect folks to decode CBOR, and then check if it is too deeply nested?

It would be easier/clearer to verify in JSON before serializing as CBOR, but that only helps with trusted PDS implementations, not federated randos or CAR imports (both on the near horizon).