[query] storing embeddings for all tweets in supabase

[srajangarg]

apart from the raw data itself, is there a plan to start creating vector embeddings for every tweet (using different embedding models) and serve it via supabase/ a cloud vector db as well?

is that under the purview of this project, or do you think individual researchers should do that themselves? in my opinion, it makes sense to reduce the inertia for people to participate in semantic research by providing standardized embeddings out of the box so people can build even faster

im asking because i was interested in making something on top of the vector embeddings and thought it makes sense to do this on top of this repo directly

[srajangarg]

just saw - https://github.com/DefenderOfBasic/twitter-semantic-search from @DefenderOfBasic - why not make the embeddings part of this project as well?

[TheExGenesis]

agreed that we want this, it's mostly a matter of it being a mess - supabase recommends having multiple projects with separate databases and I just haven't had time to set it up or find a simpler alternative that scales