My feed occasionally misses posts, not sure why

[venteto]

I have a pre-existing curated user list which was created directly in the Bluesky client, which just fetches posts from the Guggenheim, MoMA, and Tate art bots.

Now I have also created a custom feed to mimic the curated user list (and perhaps hopefully replace it). I can compare the output of my feed to that of the list, to see how well my feed is or is not working.

This is the Python snippet for my data filter for this specific feed:

        museums = [
            'did:plc:jx26hj3zymmn53cfjetkmwry',  # guggenheim
            'did:plc:pa2epocjfar2fzz4symzmb5t'   # tate
            'did:plc:z5pxuiispc5yacvofktometn',  # moma
        ]
        if author_did in museums:
            post_dict = {
                'feed_name': os.environ.get('MU_SLUG', None),
                'uri': created_post['uri'],
                'cid': created_post['cid'],
            }
            posts_to_create.append(post_dict)

This is the extent of my gunicorn config file that is serving the app on Fly (I don't have the data stream indexer and the feeds server separated, I'm not sure how or if it would even make a difference):

bind = '0.0.0.0:8000'
threads = 4
#workers = 1  # 2
#timeout = 120

But looking at the screenshot below, you can see my feed (on the right) is missing some of the posts caught by the curated list (on the left ... an example of a post my feed missed is circled in green):

It worries me that my feed is already missing posts, before I even try to get it to compare post author DID's to a list of DID's stored in a database table (for testing here, I just manually hard-coded three DID's in a Python list directly in the source code, which is not my long-term intent).

(Note: I'm not trying to resolve DID's in the Flask app, I have done that separately in an offline Django app that is storing info for accounts ... ideally in the future I want the Flask app to be able to just read from the separate Django SQLite database.)

[MarshalX]

Your provided code looks fine but thread=4 could be problematic. There is no sense in listening to the network`s event 4 times. It's hard to say why it lost something... I can only guess:

1. you work with the firehose cursor poorly
2. your firehose client reconnects without restoring from the same place (about cursors too)
3. you have some race conditions because of thread count that could lead to some undefined behavior

[venteto]

I tried dropping it down to 1 thread and 1 worker but it started missing even more posts. Now I'm trying 2 workers and 2 threads, but it's already missed one post so far. I made a repo here for the complete source code, if you wouldn't mind looking at it and pointing out any specific places I could improve, maybe how to run the firehose separately for example, if you think that would help me stop missing posts.

[MarshalX]

You can use this repo as example https://github.com/bluesky-astronomy/astronomy-feeds

[venteto]

I am closing this for now as it seems my Bluesky (bsky.app) client language settings are causing this problem