After migration to non-archival relay, not ingesting records from some independent PDSes #3094
-
|
Hi all! Congrats on keeping the ship afloat during last week's relay migration and firefighting (#3036, more). Hope you all have recovered and gotten some sleep! Since then, some independent PDSes' records are getting ingested and visible in the main appview and bsky.app, but not all. Specifically Bridgy Fed's (atproto.brid.gy) records stopped getting ingested roughly four days ago, 11/20 21:28 UTC, atproto.brid.gy subscribeRepos seq 4317000 ish. Tracked on our side in snarfed/bridgy-fed#1520 (comment). I don't know if this is happening to other independent PDSes too, or just us. Example missing record: I've talked to a few people on the team about this on and off; filing here just to have a shared place to track. Lmk if I can help. #hugops! |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 9 replies
-
|
A number of things went wrong on 2024-11-21, which impacted both Bluesky-hosted PDS instances and external PDS instances alike. I'm not aware of any current problems with new events and records being emitted in the past 24 hours, from any PDS instances in the network. Are you seeing any problems of that type, or only broken/missing content from last week? If you have a set of accounts (DIDs) missing prior content, we have a mechanism to re-index things at the AppView. A hack to get things to re-index more broadly is to emit updates or create/delete cycles on individual records. This might not make sense for posts, but has helped a few folks get profile updates unblocked (eg, updates to add and then remove whitespace from profile descriptions). |
Beta Was this translation helpful? Give feedback.
-
@bnewbold oh interesting! Thanks for the nudge, that explanation makes sense given the switch to non-archival, but atproto.brid.gy does still seem stuck somewhat PDS-wide. I tried a few different types of events and records just now, and I'm not seeing any of them:
|
Beta Was this translation helpful? Give feedback.
-
|
I am seeing some new posts now! Eg https://bsky.app/profile/vg.no , https://bsky.app/profile/bkkraindarbot.misskey.winscloud.net.ap.brid.gy Not all, though, and not account deactivations or new accounts. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, this is news to us, possibly a miscommunication on our part, we thought this was all resolved last Friday afternoon. Will prioritize investigating these examples. |
Beta Was this translation helpful? Give feedback.
-
|
I think we're properly kicked into gear now, and should be fully up to date within a few hours if all goes to plan. The ingestion rate was essentially too slow and ended-up continuously moving forward but falling behind. Will be monitoring it closely today 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @devinivy! I'm seeing new records take effect now too. Great news! Do you all know the rough time period of past records that still aren't ingested? I'd like to be able to communicate to my users, and especially to know which opt-out requests I need to redo since the account deactivations didn't take effect. |
Beta Was this translation helpful? Give feedback.
-
|
Here's the relevant timeline I was able to piece together on this (times are in EST):
Based on this, the impact that I would anticipate is that bridgy events are behind "live" to various degrees from Weds 20th 04:30PM until Mon 25th 02:30PM. Between Thurs 21st 07:45PM and 09:20PM firehose consumers such as bridgy would not have observed account events (e.g. deactivations) from bsky.network, though Bluesky core services did observe them. This does leave a hole in my understanding since you are able to cite at least this post which is not indexed by the Bluesky app. I thought we were able to maintain continuity from bridgy, and in total we've processed the number of events I would expect from bridgy since the cutover to na-relay, despite the intermittent rate limiting or disconnection. Is it possible we fell out of bridgy's backfill window, causing a gap? I will speak to the team about it and see if I could have missed something. Very much appreciate your patience here. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the details @devinivy! That all makes sense, both your timeline and the gap in understanding so far. Definitely possible that you all fell out of BF's rollback window at some point, it was only 50k seqs for a while, which was a bit less than a day. I bumped it up to 200 seqs at roughly 11/22 3p PT; I can extend it further if it'll help you all backfill. |
Beta Was this translation helpful? Give feedback.
-
|
is there anything we can do to help ? this is a follow attempt from yesterday |
Beta Was this translation helpful? Give feedback.
-
|
Some more test data from an account that's not very busy, in case it will come in useful: I signed up with https://ailbhean.co-shaoghal.net/@gunchleoc 3 days ago and I got the notification from Bridgy immediately, but it took a long while for the profile to be created at https://bsky.app/profile/gunchleoc.ailbhean.co-shaoghal.net.ap.brid.gy - might have been a few hours if I remember correctly. Today, https://bsky.app/profile/angeidheal.bsky.social followed me back as @/@[email protected] and it went through without a hitch. He hasn't posted since then, so no test data there. Most of my posts are still missing on BlueSky though. |
Beta Was this translation helpful? Give feedback.
-
|
My posts have arrived now |
Beta Was this translation helpful? Give feedback.
-
|
I think that new relay have a 2 days of lag of indexing PDS |
Beta Was this translation helpful? Give feedback.
I think we're properly kicked into gear now, and should be fully up to date within a few hours if all goes to plan. The ingestion rate was essentially too slow and ended-up continuously moving forward but falling behind. Will be monitoring it closely today 👍