Bug with retention cohorts & filtering

[paolodamico]

Bug description & expected behavior

I stumbled across this bug, when doing retention analysis. Graph 1 is the same as Graph 2 except that Graph 2 introduces an extra filter. However, as you can see from the screenshots below, for the weeks of Oct 24 & Oct 31, the cohort sizes are larger for Graph 2, which doesn't make sense because Graph 2 adds a constraint. At most, Graph 2's cohort sizes should equal Graph 1's.

Graph 1.

Graph 2.

How to reproduce

1. Open the graphs referenced above.

Environment

Cloud (Clickhouse).

Additional context

Thank you for your bug report – we love squashing them!

[EDsCODE]

This is a really interesting catch.

I think technically speaking this is possible because what's happening is that when you do first time retention it's trying to calculate the first time a user performed that event with those filters across all time. So the extra users in the graph with more filters happens because those users might actually exist already and were grouped into previous cohorts in graph 1, but then their first event with the new filter is in a later cohort so there ends up being more in those later cohorts. lmk if this makes sense!

Nontechnically speaking, this is very counter intuitive and somehow we need to clarify what we're going for here

[EDsCODE]

@paolodamico bump

[paolodamico]

Right, the technical part makes total sense. I think we can do a few things to clarify the UX:

• On the RHS panel (where you select first time retention & the filters) we could add a legend that says something like "Filtering for users that did X action with the selected filters, for the first time".
• When hovering on the cohort size we can show a tooltip with the same information (maybe emphasize the filtering part, or even better would be listing the filters in a human-friendly way).
• When opening the persons modal (after clicking a cohort), show the same information in the header of the table.

As for the goal of this query, I think that's still a bit tricky. Basically what I wanted to answer is if users who have that feature flag active (or inactive) retain better; which means I was expecting to have the same retention table, but adding a filter on top for users with the feature flag. If the cohorts are created based on when the user first performed the action while having the feature flag active, then there's the noise of when the feature flag became active for that user, and not a very accurate retention reading. Thoughts?

[posthog-bot]

This issue hasn't seen activity in two years! If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[posthog-bot]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.