Add mobile screen engagement calculation using the screen summary con…

…text (close #16)

PR #17

CHANGELOG

@@ -3,6 +3,7 @@ snowplow-unified 0.2.0 (2024-0X-XX)
 ## Summary
 
 ## Features
+- Add mobile screen engagement calculation using the screen summary context (#16)
 - Adds user stitching to the users table (enabled with `snowplow__session_stitching`)
 - Adds "headset" to the list of recognized platforms
 
@@ -13,6 +14,9 @@ snowplow-unified 0.2.0 (2024-0X-XX)
 ## Upgrading
 Bump the snowplow-unified version in your `packages.yml` file.
 
+## 🚨 Breaking Changes 🚨
+Existing users on Snowflake / Databricks / Redshift will need to make changes to some of their derived tables. For a full sql script on how to achieve this, check out the relevant [migration guide](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/migration-guides/unified/). The other option is to do a [complete refresh](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/full-or-partial-refreshes/#complete-refresh-of-snowplow-package) of the package.
+
 snowplow-unified 0.1.2 (2023-11-23)
 ---------------------------------------
 ## Summary

dbt_project.yml

@@ -85,6 +85,7 @@ vars:
     snowplow__enable_application_context: false
     snowplow__enable_screen_context: false
     snowplow__enable_deep_link_context: false
+    snowplow__enable_screen_summary_context: false
     # add extra custom fields:
     snowplow__page_view_passthroughs: []
     snowplow__session_passthroughs: []
@@ -125,6 +126,7 @@ vars:
     snowplow__application_error_events: com_snowplowanalytics_snowplow_application_error_1
     snowplow__screen_view_events: com_snowplowanalytics_mobile_screen_view_1
     snowplow__deep_link_context: com_snowplowanalytics_mobile_deep_link_1
+    snowplow__screen_summary_context: com_snowplowanalytics_mobile_screen_summary_1
 
 # Completely or partially remove models from the manifest during run start.
 on-run-start:

docs/markdown/snowplow_unified_common_cols.md

@@ -416,10 +416,14 @@ The page’s character encoding e.g. , ‘UTF-8’
 
 {% docs col_doc_width %}
 The page’s width in pixels e.g. 1024
+
+On mobile, it is the content width reported in the `screen_summary` context.
 {% enddocs %}
 
 {% docs col_doc_height %}
 The page’s height in pixels e.g. 3000
+
+On mobile, it is the content height reported in the `screen_summary` context.
 {% enddocs %}
 
 {% docs col_tr_currency %}
@@ -788,10 +792,6 @@ First application version.
 Last application version.
 {% enddocs %}
 
-{% docs col_session_duration_s %}
-Total duration of a session in seconds.
-{% enddocs %}
-
 {% docs col_device_user_id %}
 Unique device user id.
 {% enddocs %}
@@ -816,10 +816,6 @@ Earliest timestamp for the user's activity, based on `derived_tstamp`.
 Latest timestamp for the user's activity, based on `derived_tstamp`.
 {% enddocs %}
 
-{% docs col_sessions_duration_s %}
-Total session duration for the specific user.
-{% enddocs %}
-
 {% docs col_active_days %}
 Total number of active days for the user.
 {% enddocs %}
@@ -1217,3 +1213,125 @@ Referrer URL, source of this deep-link.
 {% docs col_event_index_in_session %}
 A session index of the event.
 {% enddocs %}
+
+{% docs col_foreground_sec %}
+Time in seconds spent on the current screen while the app was in foreground.
+{% enddocs %}
+
+{% docs col_background_sec %}
+Time in seconds spent on the current screen while the app was in background
+{% enddocs %}
+
+{% docs col_last_item_index %}
+Index of the last viewed item in the list on the screen
+{% enddocs %}
+
+{% docs col_items_count %}
+Total number of items in the list on the screen
+{% enddocs %}
+
+{% docs col_min_x_offset %}
+Minimum horizontal scroll offset on the scroll view in pixels
+{% enddocs %}
+
+{% docs col_max_x_offset %}
+Maximum horizontal scroll offset on the scroll view in pixels
+{% enddocs %}
+
+{% docs col_min_y_offset %}
+Minimum vertical scroll offset on the scroll view in pixels
+{% enddocs %}
+
+{% docs col_max_y_offset %}
+Maximum vertical scroll offset on the scroll view in pixels
+{% enddocs %}
+
+{% docs col_content_width %}
+Width of the scroll view in pixels
+{% enddocs %}
+
+{% docs col_content_height %}
+Height of the scroll view in pixels
+{% enddocs %}
+
+{% docs col_last_list_item_index %}
+Index of the last viewed item in the list on the screen
+
+This is calculated only for mobile apps based on the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_list_items_count %}
+Total number of items in the list on the screen
+
+This is calculated only for mobile apps based on the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_list_items_percentage_scrolled %}
+Percentage of the list on the screen that the user scrolled to.
+
+This is calculated only for mobile apps based on the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_engaged_time_in_s %}
+Time spent by the user on the page or screen.
+
+On Web, it is calculated using page pings.
+On mobile, it is calculated using information in the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_session_engaged_time_in_s %}
+The total time engaged by a user within a session.
+
+On Web, it is calculated using page pings.
+On mobile, it is calculated using information in the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_user_engaged_time_in_s %}
+The total engaged time in seconds by the user.
+
+On Web, it is calculated using page pings.
+On mobile, it is calculated using information in the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_absolute_time_in_s %}
+Total time in seconds of the page or screen view (including inactivity).
+
+On Web, it is the time between the `start_tstamp` and `end_tstamp` of the page view and the last page ping.
+On mobile, it is the time that the app was in foreground + background during the screen view (taken from the `screen_summary` context).
+{% enddocs %}
+
+{% docs col_session_absolute_time_in_s %}
+The time in seconds between the `start_tstamp` and `end_tstamp` of the first and last event in the session.
+{% enddocs %}
+
+{% docs col_user_absolute_time_in_s %}
+The time in seconds between the `start_tstamp` and `end_tstamp` of the first and last event of sessions of the user.
+{% enddocs %}
+
+{% docs col_horizontal_pixels_scrolled %}
+Distance the user scrolled horizontally in pixels.
+
+On Web, it is calculated based on the page ping events.
+On mobile, it is calculated using the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_vertical_pixels_scrolled %}
+Distance the user scrolled vertically in pixels.
+
+On Web, it is calculated based on the page ping events.
+On mobile, it is calculated using the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_horizontal_percentage_scrolled %}
+Percentage of page scrolled horizontally.
+
+On Web, it is calculated based on the page ping events.
+On mobile, it is calculated using the `screen_summary` context.
+{% enddocs %}
+
+{% docs col_vertical_percentage_scrolled %}
+Percentage of page scrolled vertically.
+
+On Web, it is calculated based on the page ping events.
+On mobile, it is calculated using the `screen_summary` context.
+{% enddocs %}

docs/markdown/snowplow_unified_views_docs.md

@@ -23,3 +23,16 @@ This model calculates the time a visitor spent engaged on a given page view. Thi
 This model calculates the horizontal and vertical scroll depth of the visitor on a given page view. Such metrics are useful when assessing engagement on a page view.
 
 {% enddocs %}
+
+{% docs table_screen_summary_metrics %}
+
+This model calculates screen engagement statistics based on the screen summary context entity tracked on mobile apps.
+It contains metrics related to the screen time and scroll depth.
+
+{% enddocs %}
+
+{% docs table_session_screen_summary_metrics %}
+
+This model calculates screen time metrics per session based on the screen summary context entity tracked on mobile apps.
+
+{% enddocs %}

integration_tests/.scripts/integration_test.sh

@@ -52,7 +52,7 @@ for db in ${DATABASES[@]}; do
 
   echo "Snowplow unified integration tests: Test models"
 
-  eval "dbt test --exclude snowplow_unified_web_vital_measurements snowplow_unified_web_vital_measurements_actual snowplow_unified_web_vital_events_this_run test_name:not_null --store-failures --target $db" || exit 1;
+  eval "dbt test --exclude snowplow_unified_web_vital_measurements snowplow_unified_web_vital_measurements_actual snowplow_unified_web_vital_events_this_run snowplow_unified_views_mobile_screen_engagement_actual test_name:not_null --store-failures --target $db" || exit 1;
 
   echo "Snowplow unified integration tests: All non-CWV tests passed"
 
@@ -72,4 +72,12 @@ for db in ${DATABASES[@]}; do
 
   echo "Snowplow unified integration tests: All CWV tests passed"
 
+  echo "Snowplow unified integration tests: Test mobile screen engagement"
+
+  eval "dbt run --select +snowplow_unified_views_mobile_screen_engagement_actual snowplow_unified_views_mobile_screen_engagement_expected_stg source --full-refresh --vars '{snowplow__allow_refresh: true, snowplow__start_date: '2023-12-19', snowplow__backfill_limit_days: 50, snowplow__enable_cwv: false, snowplow__enable_screen_summary_context: true, snowplow__enable_ua: false, snowplow__enable_iab: false, snowplow__enable_web: false, snowplow__enable_browser_context: false, snowplow__enable_consent: false, snowplow__enable_yauaa: false, snowplow__enable_geolocation_context: false, snowplow__enable_deep_link_context: false, snowplow__enable_app_errors: false}' --target $db" || exit 1;
+
+  eval "dbt test --select snowplow_unified_views_mobile_screen_engagement_actual --vars '{snowplow__enable_screen_summary_context: true, snowplow__enable_web: false, snowplow__enable_cwv: false, snowplow__enable_ua: false, snowplow__enable_iab: false, snowplow__enable_web: false, snowplow__enable_browser_context: false, snowplow__enable_consent: false, snowplow__enable_yauaa: false, snowplow__enable_geolocation_context: false, snowplow__enable_deep_link_context: false, snowplow__enable_app_errors: false}' --store-failures --target $db" || exit 1;
+
+  echo "Snowplow unified integration tests: Mobile screen engagement tests passed"
+
 done

integration_tests/README.md

@@ -39,7 +39,7 @@ There are certain exceptions to how different warehouses process data and in pla
 
 - the non-deterministic nature of row_number() function for Redshift/Postgres/Databricks means that we had to hard-code actuals and expected models for cases where we are testing duplicate rows with exact same results / window
 - postgres / redshift needing the array format of : (within sessions_expected)
-- bigquery handling of snowplow_utils.timestamp_diff() - absolute_time_in_s changes as well as sessions_duration_s
+- bigquery handling of snowplow_utils.timestamp_diff() - absolute_time_in_s changes
 - rotating domain_userid per session is hard-coded in the integration test expectations, when run in one batch the user_identifier differs: 2e340eb6e94820ea8369c0174c612260d1cfe9d41f0fe46268994e28d9c0bbf17
 0e9ab97b5d9d9a174112df13fe9c44788af3ac9088a8b41e0998d92a8b4b5a4fc
 - same with the number of quarantined sessions