Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

Closes #11909: Metrics for recording number of recently used PWAs #11982

Merged

Conversation

grigoryk
Copy link
Contributor

@grigoryk grigoryk commented Jun 26, 2020

Requires mozilla-mobile/android-components#7502 and mozilla-mobile/android-components#7519

We can't count exact installs (Android doesn't inform us about their deletions), so instead we look at number of "recently used" (within last 30 days) as a proxy value.

Pull Request checklist

  • Tests: This PR includes thorough tests or an explanation of why it does not
  • Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
  • Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

After merge

  • Milestone: Make sure issues finished by this pull request are added to the milestone of the version currently in development.

To download an APK when reviewing a PR:

  1. click on Show All Checks,
  2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
  3. click on the "Fenix - assemble" task, then click "Run Artifacts".
  4. the APK links should be on the left side of the screen, named for each CPU architecture

@grigoryk
Copy link
Contributor Author

Will fill-out a data review once I confirm this is all good and working as expected.

@grigoryk grigoryk force-pushed the issue11909PwaInstallsTelemetry branch 2 times, most recently from a0995b0 to e8b487d Compare June 26, 2020 01:14
@grigoryk
Copy link
Contributor Author

Confirmed that this does work, but needs mozilla-mobile/android-components#7519 to land first.

@grigoryk
Copy link
Contributor Author

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

  1. What questions will you answer with this data?

We want to know if our users are actually using PWAs, and if so how many.

  1. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

These counts are crucial in understand impact of the PWA functionality, and helpful in understanding other browser usage in context of how this feature is used.

  1. What alternative methods did you consider to answer these questions? Why were they not sufficient?

We are also working on event telemetry for PWAs for more detailed usage information, but it won't be a substitute for this simple high-level metric.

  1. Can current instrumentation answer these questions?

No.

  1. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

Measurement Description Data Collection Category Tracking Bug #
recently_used_pwa_count Category 2 #11909
has_recent_pwas Category 2 #11909
  1. Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

This collection is documented in the metrics.md document

  1. How long will this data be collected? Choose one of the following:

expires: "2020-12-01"

  1. What populations will you measure?

All channels/locales.

  1. If this data collection is default on, what is the opt-out mechanism for users?

Standard Glean opt-out.

  1. Please provide a general description of how you will analyze this data.

Through a re-dash dashboard.

  1. Where do you intend to share the results of your analysis?

Within Fenix/A-C teams.

  1. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? If so:

No.

@grigoryk grigoryk requested a review from liuche June 26, 2020 01:33
@grigoryk grigoryk added the needs:data-review PR is awaiting a data review label Jun 26, 2020
@grigoryk grigoryk requested a review from boek June 26, 2020 01:33
Copy link
Contributor

@liuche liuche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data Review Form (to be filled by Data Stewards)

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, documented in metrics.md

  1. Is there a control mechanism that allows the user to turn the data collection on and off?

controlled by Fenix data controls

  1. If the request is for permanent data collection, is there someone who will monitor the data over time?

Expiry 12/1/2020

  1. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 2, usage of PWA feature

  1. Is the data collection request for default-on or default-off?

default on

  1. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No, usage counter

  1. Is the data collection covered by the existing Firefox privacy notice?

yes

  1. Does there need to be a check-in in the future to determine whether to renew the data? (Yes/No) (If yes, set a todo reminder or file a bug if appropriate)**

No, expiry will kick in

  1. Does the data collection use a third-party collection tool? If yes, escalate to legal.

No

app/metrics.yaml Outdated Show resolved Hide resolved
app/metrics.yaml Show resolved Hide resolved
@@ -523,6 +527,23 @@ open class HomeActivity : LocaleAwareAppCompatActivity() {
this.visualCompletenessQueue = visualCompletenessQueue
}

private fun captureSnapshotTelemetryMetrics() = CoroutineScope(Dispatchers.IO).launch {
// PWA
val recentlyUsedPwaCount = components.core.webAppShortcutManager.recentlyUsedWebAppsCount(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what a convenient call, is it just for telemetry?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup.

@grigoryk grigoryk force-pushed the issue11909PwaInstallsTelemetry branch from e8b487d to 9b198e1 Compare June 26, 2020 01:55
@liuche liuche removed the needs:data-review PR is awaiting a data review label Jun 26, 2020
@grigoryk grigoryk requested review from eliserichards and removed request for boek June 26, 2020 02:07
Copy link

@eliserichards eliserichards left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I have a few nits 😄

app/metrics.yaml Show resolved Hide resolved
app/src/main/java/org/mozilla/fenix/HomeActivity.kt Outdated Show resolved Hide resolved
@grigoryk grigoryk force-pushed the issue11909PwaInstallsTelemetry branch from 9b198e1 to f008d75 Compare June 26, 2020 18:06
@grigoryk grigoryk requested a review from eliserichards June 26, 2020 18:07
Copy link

@eliserichards eliserichards left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🥇

@grigoryk grigoryk force-pushed the issue11909PwaInstallsTelemetry branch 2 times, most recently from d218025 to 4ecc9d3 Compare June 26, 2020 20:34
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants