Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extract funnel steps directly in the events subquery #13922

Open
yakkomajuri opened this issue Jan 25, 2023 · 2 comments
Open

Extract funnel steps directly in the events subquery #13922

yakkomajuri opened this issue Jan 25, 2023 · 2 comments
Assignees
Labels
feature/funnels Feature Tag: Funnels performance Has to do with performance. For PRs, runs the clickhouse query performance suite team/product-analytics

Comments

@yakkomajuri
Copy link
Contributor

Currently we pass all of an event's properties from the innermost funnels subquery to be parsed into the steps further up. The problem with this is that we then pipe all that data over the network from one shard to another, whereas we could push a lot of the computations down to the individual shards and only send the results over.

I've done some initial benchmarking on this and the results are promising:

https://metabase.posthog.net/dashboard/24-query-benchmarking?query_tag=funnels-84510088626212
Screenshot 2023-01-25 at 14 44 26

https://metabase.posthog.net/dashboard/24-query-benchmarking?query_tag=funnels-29424020244

Screenshot 2023-01-25 at 14 44 36

Just look at the difference in cross-shard data transfer!

The change is reasonably simple from the perspective of the final query, but it's a bit of an involved refactor given how complicated the funnels query building logic is.

Writing this issue down to document things but I'm already working on it.

Another thing is that I think we can also remove a few levels of nesting when it comes to breakdowns, but that's something I need to also dig into.

@yakkomajuri yakkomajuri self-assigned this Jan 25, 2023
@yakkomajuri yakkomajuri added team/product-analytics performance Has to do with performance. For PRs, runs the clickhouse query performance suite feature/funnels Feature Tag: Funnels labels Jan 25, 2023
@mariusandra
Copy link
Collaborator

Just double checking, will this affect things the hogql property filters PR touches somehow? I remember adding some special handling for hogql property filters in funnels, though since materialised properties work (and are passed outside properties obviously), I expect there to not really be any issues 🤔

Getting the HogQL PR in will be my nr 1 priority tomorrow morning...

@yakkomajuri
Copy link
Contributor Author

yakkomajuri commented Jan 26, 2023

I believe it shouldn't no, but will keep an eye as I make changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/funnels Feature Tag: Funnels performance Has to do with performance. For PRs, runs the clickhouse query performance suite team/product-analytics
Projects
None yet
Development

No branches or pull requests

2 participants