feat: support for conversation_id

[ri72miieop]

This PR adds the field conversation_id to the tweets table. It has a value when the whole chain of replies up to the main tweet is found on the archive, otherwise it stays NULL.

When the user finishes the upload of the archive and the upload_stage is changed to 'complete' a new job with the key 'update_conversation_ids' is queued. This job calls a function which runs the update over all the tweets (this is needed to also update conversation_id of other users that have interacted with tweets from the one who uploaded, but there's some space for optimizations here) and refreshes the materialized view.

The materialized view 'main_thread_view' contains only the tweets in the main thread, so no replies from other users nor a continuation of a conversation between the user who wrote the thread and others.

related to #70

[vercel]

@ri72miieop is attempting to deploy a commit to the theexgenesis' projects Team on Vercel.

A member of the Team first needs to authorize it.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
community-archive	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 28, 2024 8:00pm

[TheExGenesis]

@ri72miieop you can run pnpm build locally to check if it'll build in production - in this case it was this error

[TheExGenesis]

btw for long running materialized view refreshes it's best to do it concurrently - otherwise selects are prevented REFRESH MATERIALIZED VIEW CONCURRENTLY main_thread_view;

[TheExGenesis]

I also wouldn't put conversation_id in the main tweets table, I'd make a separate table, just bc it's something we're adding on

[TheExGenesis]

I would also put all these functions in the private schema, the only reason to be in public is if we need to call them from the client side, as a user

[TheExGenesis]

I'm also wondering if that materialized view is necessary. Do you have a specific purpose in mind for it? Might it be enough to have a regular view that computes just in time?

[TheExGenesis]

for context here

[ri72miieop]

Changes made:

• Move conversation_id from tweets table to a new table called conversations
• Change materialized view to calculate info for all threads for a function that does that same for a single conversation_id
• Create view tweets_w_conversation_id

It's compiling on my machine but it seems to be failing on Vercel? Not sure if that's because there's some manual process necessary or it is failing to compile, if there's any issue let me know.

[TheExGenesis]

@ri72miieop Thanks for updating! Looping over individual tweets seems inefficient, have you considered starting from root tweets and processing them in layers? I asked Claude to draft a proposal. Haven't tested yet.

[ri72miieop]

Yeah this is similar to what I was doing before. I tested it now, results:

Going one by one seems inefficient but I am taking advantage of the order of the tweets, so when I search for the tweet with the id reply_to_tweet_id in the temporary table I have a guarantee that it either is there and already has the conversation_id computed or is null which means we don't have the complete thread in the db yet so we can ignore for now.

Meanwhile in the recursive case if it is a thread with like 300 elements it will have to recursively search many times for each of the tweet (this is why I it was taking 1m with my test data that had 10% of the tweets of the archive and with the full archive it was timing out).

my SQL is rusty, so if I missed something let me know, but I think this is the reason recursively it scales so much worse with more data than just going through each one of them one by one

[TheExGenesis]

testing the latest functions
btw there's a typo in conversations table, needs a comma after "text"

CREATE TABLE IF NOT EXISTS "public"."conversations" (
    "tweet_id" text NOT NULL PRIMARY KEY,
    "conversation_id" text,
    FOREIGN KEY (tweet_id) REFERENCES public.tweets(tweet_id)
);