Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a unified Snuba consumer #1670

Closed
Tracked by #232 ...
BYK opened this issue Feb 11, 2021 · 4 comments
Closed
Tracked by #232 ...

Create a unified Snuba consumer #1670

BYK opened this issue Feb 11, 2021 · 4 comments
Labels
enhancement New feature or request wontfix This will not be worked on

Comments

@BYK
Copy link
Member

BYK commented Feb 11, 2021

Currently we have many Snuba consumers and seems like we are going to add more as we add more features around Snuba. This poses a maintenance and synchronization issue for on-premise as the growing number of consumers is hard to maintain. People also keep forgetting to add new consumers to the self-hosted repo, causing issues or missing features (this has happened twice now).

Proposal: introduce a consumers command to Snuba CLI that runs all core consumers with default settings (the settings we use in self-hosted) using multiprocessing so the consumers are still logically separate.

/cc @fpacifici @lynnagara @chadwhitacre

@BYK
Copy link
Member Author

BYK commented Mar 1, 2023

84years

@chadwhitacre
Copy link
Member

Here it is @JoshFerge. :)

@volokluev
Copy link
Member

This is too much work that our team does not have time for at the moment.

@untitaker
Copy link
Member

I think we should eventually do that but it's too early to create tickets. I think with lightweight deletes, EAP mutations, subscriptions the code became a lot more duplicated, the friction for creating new consumers in snuba feels very high, and the refactor in this issue would solve it. we did a similar project in sentry where we have way more consumers and it took me less than a quarter. I think it can be done in a limited fashion for rust only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

4 participants