Pre-compute safe dimension slices

[lukesonnet]

This PR introduces "Dimension Metadata", which uses existing Experiment Dimensions and creates stable bins that can be used to reliably only get a max of 21 (20 + "other") bins per experiment dimension. This helps deal with the problem where dimensions are high cardinality and are therefore unsuitable to pre-compute health queries or analysis queries for dimensions with every update.

This PR has a three key components:

• back-end model and API endpoints
  • A new model DimensionMetadataModel to store dimension results and query data
  • New CRUD endpoints for dimension metadata in the datasources controller
  • Changed the ExposureQuery interface to store the id of the latest "saved" automatic dimension as well as the computed results from the interface with the dimension metadata query runner
• back-end SQL and query runner changes
  • A new DimensionMetadataQueryRunner that can manage running the query to pick the slices using historical data from an experiment exposure query
  • Changes to ExperimentResultsQueryRunner to use dimension metadata when needed (right now just in the units query and in the traffic query)
  • In SQLIntegration.ts, two key changes:
    • the new getDimensionMetadataQuery that scans the last 30 days of your exposure query and gets the top 20 dimensions for each experiment dimension
    • changes to the experiment units and traffic queries to bucket experiment dimensions using pre-existing buckets, if they exist (allowedValues) to reduce cardinality
• Front-end modal
  • A modal at the exposure query level to allow users to run the dimensions metadata and save the id along with the computed results to the ExposureQuery if they wish to use it later

TODO (maybe in fast follow):

• Smaller entry point to modal for V0 (DONE: in kebab menu)
• Better messaging when 0 rows return
• Better updating of the automatic exposures when the exposure query or its dimensions change (later, but for now at least we only show people the dimension metadata that already exists for a dimension name).
• Allow setting of number of days for lookback

Loom overview: https://www.loom.com/share/17bae04087e94f8299512d9b857b5f6b

Loom error states: https://www.loom.com/share/4382c49e44d34f9891ce81786523e097

Health query that uses automatic dimensions

-- Traffic Query for Health Tab
WITH
  __rawExperiment AS (
    SELECT
      userid as user_id,
      timestamp as timestamp,
      experimentid as experiment_id,
      variationid as variation_id,
      browser as browser,
      country as country,
      floor(random() * 100 + 1)::int as random
    FROM
      experiment_viewed
  ),
  __experimentExposures AS (
    -- Viewed Experiment
    SELECT
      e.user_id as user_id,
      cast(e.variation_id as varchar) as variation,
      e.timestamp as timestamp,
      (
        CASE
          WHEN browser IN (
            'Chrome',
            'Safari',
            'Edge',
            'Firefox',
            'Samsung Internet',
            'Opera'
          ) THEN cast(browser as varchar)
          ELSE cast('__Other__' as varchar)
        END
      ) AS dim_browser,
      (
        CASE
          WHEN country IN ('US', 'UK', 'CA', 'AU') THEN cast(country as varchar)
          ELSE cast('__Other__' as varchar)
        END
      ) AS dim_country,
      (
        CASE
          WHEN random IN (
            '77',
            '5',
            '59',
            '47',
            '68',
            '38',
            '100',
            '96',
            '43',
            '27',
            '71',
            '37',
            '19',
            '80',
            '39',
            '20',
            '58',
            '65',
            '72',
            '17'
          ) THEN cast(random as varchar)
          ELSE cast('__Other__' as varchar)
        END
      ) AS dim_random
    FROM
      __rawExperiment e
    WHERE
      e.experiment_id = 'checkout-layout'
      AND e.timestamp >= '2021-11-22 20:23:00'
      AND e.timestamp <= '2023-11-17 23:52:00'
  ),
  __experimentUnits AS (
    -- One row per user
    SELECT
      e.user_id AS user_id,
      (
        CASE
          WHEN count(distinct e.variation) > 1 THEN '__multiple__'
          ELSE max(e.variation)
        END
      ) AS variation,
      MIN(e.timestamp) AS first_exposure_timestamp,
      SUBSTRING(
        MIN(
          CONCAT(
            SUBSTRING(
              to_char(e.timestamp, 'YYYY-MM-DD HH24:MI:SS.MS'),
              1,
              19
            ),
            coalesce(
              cast(e.dim_browser as varchar),
              cast('__NULL_DIMENSION' as varchar)
            )
          )
        ),
        20,
        99999
      ) AS dim_exp_browser,
      SUBSTRING(
        MIN(
          CONCAT(
            SUBSTRING(
              to_char(e.timestamp, 'YYYY-MM-DD HH24:MI:SS.MS'),
              1,
              19
            ),
            coalesce(
              cast(e.dim_country as varchar),
              cast('__NULL_DIMENSION' as varchar)
            )
          )
        ),
        20,
        99999
      ) AS dim_exp_country,
      SUBSTRING(
        MIN(
          CONCAT(
            SUBSTRING(
              to_char(e.timestamp, 'YYYY-MM-DD HH24:MI:SS.MS'),
              1,
              19
            ),
            coalesce(
              cast(e.dim_random as varchar),
              cast('__NULL_DIMENSION' as varchar)
            )
          )
        ),
        20,
        99999
      ) AS dim_exp_random
    FROM
      __experimentExposures e
    GROUP BY
      e.user_id
  ),
  __distinctUnits AS (
    SELECT
      user_id,
      variation,
      to_char(
        date_trunc('day', first_exposure_timestamp),
        'YYYY-MM-DD'
      ) AS dim_exposure_date,
      dim_exp_browser,
      dim_exp_country,
      dim_exp_random
    FROM
      __experimentUnits
  )
  -- One row per variation per dimension slice
  -- dim_exposure_date
  (
    SELECT
      variation AS variation,
      dim_exposure_date AS dimension_value,
      MAX(cast('dim_exposure_date' as varchar)) AS dimension_name,
      COUNT(*) AS units
    FROM
      __distinctUnits
    GROUP BY
      variation,
      dim_exposure_date
  )
UNION ALL
-- dim_exp_browser
(
  SELECT
    variation AS variation,
    dim_exp_browser AS dimension_value,
    MAX(cast('dim_exp_browser' as varchar)) AS dimension_name,
    COUNT(*) AS units
  FROM
    __distinctUnits
  GROUP BY
    variation,
    dim_exp_browser
)
UNION ALL
-- dim_exp_country
(
  SELECT
    variation AS variation,
    dim_exp_country AS dimension_value,
    MAX(cast('dim_exp_country' as varchar)) AS dimension_name,
    COUNT(*) AS units
  FROM
    __distinctUnits
  GROUP BY
    variation,
    dim_exp_country
)
UNION ALL
-- dim_exp_random
(
  SELECT
    variation AS variation,
    dim_exp_random AS dimension_value,
    MAX(cast('dim_exp_random' as varchar)) AS dimension_name,
    COUNT(*) AS units
  FROM
    __distinctUnits
  GROUP BY
    variation,
    dim_exp_random
)
LIMIT
  3000

Testing

To test you need the feature flag health-tab to be on for you, and to actually run the health queries with auto dimensions you need to have turned on the traffic query in your org settings.

[github-actions]

Your preview environment pr-1873-bttf has been deployed with errors.

Preview environment endpoint is available here

[bryce-fitzsimons]

can we be more specific?

[lukesonnet]

I'm just passing through the query error here, so not really. I could just have "QUERY_ERROR" and then you could look up the query error by reference in the front end, I guess?

[jdorn]

Do we need this link in the database? We already have a link the other direction since there's a data source and exposureQueryId in the automatic dimension model.

[lukesonnet]

Yes, because I don't want to automatically just get the latest value from that collection (e.g. they re-run the query but it gives them garbage) and they choose not to save it.

[jdorn]

Should there be some sanity check for when there are no dimensions?

[lukesonnet]

Yeah, good call. It currently just generates broken SQL. I've updated it to throw an error and surface it to the user. I also now hide the entry point to this modal when there are no dimensions.

[bryce-fitzsimons]

I'd probably change the name of this model to be less abstract (metadata schema) to more specific (dimension value queries, etc)

[bryce-fitzsimons]

same suggestion about naming (metadata too broad?)

[lukesonnet]

Ok, renaming to dimensionSlices where important. I still keep it as dimensionMetadata in the ExposureQueryInterface because we may add additional metadata to that object.