wip: support test_ api key

[buwilliams]

Changes

• Support for test_[apiKey] which signifies local, test, and staging environments
• Allows for filtering all data by drawing a line between Production and everything other env
• test_[apiKey] sets $environment to test
• Users may also pass in this property to override or simply set the value for $environment
• Added environment toggle to top navigation and put it behind a feature flag

See #3149 for more information

Checklist

• All querysets/queries filter by Organization, by Team, and by User
• Django backend tests
• Jest frontend tests
• Cypress end-to-end tests
• Migrations are safe to run at scale (e.g. PostHog Cloud) – present proof if not obvious

[buwilliams]

Anyone know what this check is failing? It's a python issue that doesn't make sense to me.

[paolodamico]

Re @buwilliams,

Anyone know what this check is failing? It's a python issue that doesn't make sense to me.

The check that was failing indicated that there was a typing problem. Python has only limited support for typing, introduced fairly recently (see more details). One of the limitations is that typing is not actually checked when coding (some IDEs might support this) or at runtime. Instead, we use https://github.com/python/mypy to check typing. You can run it yourself locally by doing DEBUG=1 mypy .

See 87018fc for details on how the specific typing issue was fixed.

[paolodamico]

Added some comments. Also proposing a different UI for the toggle (feel free to challenge / iterate).

1. Also worth noting, what happens when you change the filter? Ideally we would reload the graph, table you're viewing.
2. Finally, we should also remove the test filters toggle from the UI if the feature flag is enabled.

[paolodamico]

Here's my proposed UI for this. Feedback @corywatilo too?

Outstanding

Please note I think we should also add a global indicator to all pages to be more explicit when you're on test data mode. Something like this,

[Twixes]

Hey! So, I posted the original env separation issue, and I think this implementation may be too narrow. I think it's quite possible we'll want to support more such environments than with just a boolean switch (e.g. selector a la Sentry), so it'd be more flexible if instead of $test_environment: true/false we used $environment: 'production'/'development'.
Aaand, that still does not address a key issue that motivates this feature request: users want to get rid of environments with junk data, but that's not possible - events, once ingested, they history forever, because we don't delete CH events yet. With projects that's kind of solved because the data is just inaccessible once the project is deleted, thanks to using team_id for all filtering.

[buwilliams]

@Twixes Why does removing the data matter? Filtering removes unwanted results so the effect is the same from a UI perspective. Additionally, if a flag such as test_environment is present, deleting that data becomes trivial. In either case, removing the data seems like a follow-up issue. One that allows users to purge data based on a filter.

The reason for having boolean rather than by environment was to aid UX since we can think about production and all others and two distinct things and present a simple UX for it. Otherwise, we'd need to create an additional table where users would be to specify their environments so that we could group them appropriately. Seems to me, if an advanced user wanted to track per environment, they could add this as a regular event property. Something like: environment=production

[buwilliams]

I've changed the Boolean to a string and renamed it to $environment. Currently, it supports: "production" and "test". Also, I've moved it into a hidden property. Push coming soon.

[Twixes]

Yep, okay, actually removing the data from our disks doesn't matter (that much), but removing it from the user's point of view (meaning just making it completely inaccessible) matters. So it's enough to just always and everywhere filter based on this property. And the environments for a project can be just a Postgres array of strings on the Team model for now.
BTW if we introduce this, it'd be better as an actual Postgres/ClickHouse column on the events table – getting the property from JSON is significantly more expensive. Deployments & Infrastructure would be of help.

[paolodamico]

I agree with @buwilliams, I think multiple environments could be a more advanced use and to consider for later.

In terms of data deletion, I'm torn but leaning towards this is the right approach. Deleting data is a separate thing we need to tackle.

@buwilliams perhaps it's useful to add the context on why we decided against separate projects for this ?

[buwilliams]

I'm blocked until M1. What I thought was a bug in my code was not. Events are not ingested because we process events by sending them to a job queue on our plugin server. I was misguided because /demo inserts history for us which appeared as though I was receiving events. Our plugin server relies on Clickhouse which isn't support on M1 as most of you know. Maybe someone knows a way around this but I haven't been able to find it.

[mariusandra]

To clarify one point, the plugin server definitely does not rely on clickhouse, as we also use the same plugin server for the OSS / non-clickhouse installations. Without explicitly setting KAFKA_ENABLED to true, the plugin server uses the standard postgres ingestion.

[buwilliams]

Solved the issue with the plugin-server by running yarn build; WORKER_CONCURRENCY=2 yarn start:dist and created an issue #370. This is ready for merge once all the checks pass, we can continue to improve it in another PR.