Accounts on federated PDSes sometimes get "stuck" and stop updating in the appview

[snarfed]

Hi all! This has been discussed lightly in other places for a while, but I don't know if there's a dedicated discussion or issue for it. If there is, feel free to point me there instead!

For a while now in Bridgy Fed, we've seen a steady trickle of our ATProto accounts get "stuck" at some point, and records we emit for them stop appearing in bsky.app. Early on, I think one trigger here was emitting records that didn't pass lexicon validation, but I'm pretty confident we squashed that a while back, and this is still happening.

On the technical side, this is probably our top priority ask for the Bluesky team right now, since we don't have any visibility into what's happening or what we can do about it.

@bnewbold nodded to the need for tools here recently in https://bsky.app/profile/bnewbold.net/post/3lcmad4ktkk26 . https://atproto.tools/records (RIP?) was somewhat useful in the past when this was relay related, as was running my own relay locally to debug. I suspect the cases of this now are more appview related, though. (I don't have evidence, just a hunch.)

A few users recently have said this happened when they set a custom domain handle on their bridged account. That seems like an unrelated coincidence to me, but I guess it's possible?

Here are a couple examples:

• @makigas.es, https://bsky.app/profile/did:plc:xhyk5f7cpndo4lrlkylf7l4h, stopped updating 2 mos ago
  recent record: at://did:plc:xhyk5f7cpndo4lrlkylf7l4h/app.bsky.actor.profile/self, commit bafyreiekz3uorn2kbz5ynftvhuv66ampfwga27cgr7ikf6xaqihkwxzp5e, atproto.brid.gy subscribeRepos seq 5505770, emitted 2024-12-09 11:03:20 UTC
• @nosgustalinux.es, https://bsky.app/profile/did:plc:b4zgkneq5mcqm3gpmztfbm4w, stopped updating 1w ago
  recent record: at://did:plc:b4zgkneq5mcqm3gpmztfbm4w/app.bsky.feed.post/3lcukk3k5sw32, commit bafyreibgnstiaoknz6xdihalkuyatcjoj2sdl7w3luqhdzczb4nmmsxgqq , atproto.brid.gy subscribeRepos seq 5505071, emitted 2024-12-09 between 10:53:26 and 11:05:38 UTC (not sure when for your relay specifically)

Interestingly, when I try to goat get these two at:// URIs, I get:

error: XRPC ERROR 400: InvalidRequest: in com.atproto.repo.getRecord, string cid with value `''`: is invalid for format cid

...but getRecord on atproto.brid.gy returns them fine, eg https://atproto.brid.gy/xrpc/com.atproto.repo.getRecord?repo=did:plc:b4zgkneq5mcqm3gpmztfbm4w&collection=app.bsky.feed.post&rkey=3lcukk3k5sw32 , and when I paste the records and their lexicons into https://atproto.tools/lexicons , it says they validate ok.

Thank you all in advance!

[DavidBuchanan314]

Not sure if it's related to the specific issue at hand, but https://fed.brid.gy/xrpc/com.atproto.sync.getRecord?did=did:plc:b4zgkneq5mcqm3gpmztfbm4w&collection=app.bsky.feed.post&rkey=3lcukk3k5sw32 is returning a CAR with a single block (the record itself), with no MST blocks or commit object

[snarfed]

Hah, true! That's a longstanding missing bit that we need to implement: snarfed/arroba#32 . Should be unrelated here though? Last time I checked the relay doesn't use sync.getRecord anywhere in its repo synching, just subscribeRepos and getRepo.

[DavidBuchanan314]

Yeah, I don't think it should matter - I just noticed because I tried inspecting the record in pdsls.dev and it failed to verify (because pdsls relies on sync.getRecord for that)

[DavidBuchanan314]

and on re-reading the spec I realise millipds isn't doing it correctly either - I don't handle the "or non-existence" case!

[bnewbold]

Our current hypothesis for a bunch of these issues is a problem in the non-archival relay we deployed in recent weeks (after a day of multiple relay issues). The branch has not yet been merged or deployed with the most recent commits. I think you can use this as a public tracking issue (but please don't comment over there):

bluesky-social/indigo#827

[snarfed]

Wonder if this is related to #3215 at all...?

[devinivy]

During the investigation that led to the fix in #3215 I was hoping/expecting to find that bridgy and millipds were bumping into the same issue. But I actually wasn't able to turn up any accounts hosted by bridgy that were impacted by that issue.

The relay change linked above is now being canaried, and I'll be checking in to see if there are any improvements in the situation.

If you happen to have any more events that are recent (e.g. past 48 hours) which didn't seem to make it through, would be interested to check those out in particular.

[snarfed]

@devinivy thanks for looking!

I admit, I'm a bit afraid we'll eventually discover that I have a subtle occasional MST bug or similar here. Ah well, still worth figuring out sooner than later at least, if so.

Here's a recent record on a still-stuck repo: at://did:plc:b4zgkneq5mcqm3gpmztfbm4w/app.bsky.feed.post/3lcziri3fmoi2, atproto.brid.gy subscribeRepos seq 5636397, first emitted 2024-12-11 10:01:50 UTC.