Skip to content
This repository has been archived by the owner on Jan 12, 2023. It is now read-only.

Record telemetry for nimbus experiments fetching #6972

Merged

Conversation

ionutbedregeanu
Copy link
Contributor

@ionutbedregeanu ionutbedregeanu commented May 5, 2022

For #6847
Add a timespan to know how much it takes to download the experiments
No tests, screenshots, accessibility required

Pull Request checklist

  • Tests: This PR includes thorough tests or an explanation of why it does not
  • Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
  • Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

To download an APK when reviewing a PR:

  1. click on Show All Checks,
  2. click Details next to build-focus-debug or build-klar-debug for changes targeting Klar,
  3. click View task in Taskcluster,
  4. click the Artifacts row,
  5. click to download any of the apks listed here which use an appropriate name for each CPU architecture.

@ionutbedregeanu ionutbedregeanu force-pushed the ionut.bedregeanu/6847_telemetry branch from 67770a8 to 3698f01 Compare May 5, 2022 12:15
@ionutbedregeanu
Copy link
Contributor Author

ionutbedregeanu commented May 5, 2022

Request for data collection review form

All questions are mandatory. You must receive a review from a data steward peer on your responses to these questions before shipping new data collection.

1) What questions will you answer with this data?
How much it takes to fetch the nimbus experiments?

2) Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?
This is needed to analyze feature usage and possible feature improvements.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?
This is the only way.

4) Can current instrumentation answer these questions?
No. We need a new specific event for a specific app feature.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.

Measurement Description Data Collection Category Tracking Bug #
Nimbus experiments timespan Category 1 https://github.com//issues/6847

6) How long will this data be collected? Choose one of the following:
Expiry for capturing data is set to release version 112

7) What populations will you measure?
All release channels and locales.

8) If this data collection is default on, what is the opt-out mechanism for users?
Users can opt out of data collection by disabling Usage and technical data from Settings -> Privacy and security -> Data choices.

9) Please provide a general description of how you will analyze this data.
Glean

10) Where do you intend to share the results of your analysis?
Only on Glean, mobile teams, and BD have internal access.

12) Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?
No.

@ionutbedregeanu ionutbedregeanu marked this pull request as ready for review May 5, 2022 12:56
@ionutbedregeanu ionutbedregeanu added the 🕵️‍♀️ needs review PRs that need to be reviewed label May 5, 2022
@ionutbedregeanu ionutbedregeanu marked this pull request as draft May 6, 2022 12:13
@ionutbedregeanu ionutbedregeanu removed the 🕵️‍♀️ needs review PRs that need to be reviewed label May 6, 2022
@ionutbedregeanu ionutbedregeanu force-pushed the ionut.bedregeanu/6847_telemetry branch from 3698f01 to 7047ff5 Compare May 9, 2022 06:44
@ionutbedregeanu ionutbedregeanu marked this pull request as ready for review May 9, 2022 06:45
@ionutbedregeanu ionutbedregeanu added the 🕵️‍♀️ needs review PRs that need to be reviewed label May 9, 2022
@ionutbedregeanu
Copy link
Contributor Author

ionutbedregeanu commented May 9, 2022

successful_fetch should be renamed and converted to an event which should contains information about the state(failed or not), but also about the internet connection(if the device is connected to internet or not).
@Amejia481 @jhugman What do you think about this?

@ionutbedregeanu ionutbedregeanu removed the 🕵️‍♀️ needs review PRs that need to be reviewed label May 9, 2022
@ionutbedregeanu ionutbedregeanu marked this pull request as draft May 9, 2022 12:59
@Amejia481
Copy link
Contributor

successful_fetch should be renamed and converted to an event which should contains information about the state(failed or not), but also about the internet connection(if the device is connected to internet or not). @Amejia481 @jhugman What do you think about this?

I think a boolean it's sufficient as long as we report true when we are able to fetch the experiments and false otherwise.

Copy link
Contributor

@jhugman jhugman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced of the utility of the successfulFetch event, so suggest you remove it.

Also: could you amend the data review request to make the data category "Category 1: Technical". Edit: I see you've done the right thing in the metrics.yaml!

Otherwise, this is nice, well done. Thank you.

@ionutbedregeanu ionutbedregeanu marked this pull request as ready for review May 10, 2022 11:03
@ionutbedregeanu ionutbedregeanu added the 🕵️‍♀️ needs review PRs that need to be reviewed label May 10, 2022
…ching

Add a timespan to know how much it takes to download the experiments
@ionutbedregeanu ionutbedregeanu force-pushed the ionut.bedregeanu/6847_telemetry branch from 7047ff5 to deef6ff Compare May 10, 2022 12:20
@Amejia481 Amejia481 self-requested a review May 10, 2022 13:51
Copy link
Contributor

@Amejia481 Amejia481 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having data to set a good default was the principal point, but we also wanted to have an overview idea of how many users could get/fail to fetch based on #6847 requirements. Let's not block this we could do a follow up if it's needed.

@ionutbedregeanu ionutbedregeanu added 🛬 needs landing PRs that are ready to land and removed 🕵️‍♀️ needs review PRs that need to be reviewed labels May 12, 2022
@mergify mergify bot merged commit 48f42a8 into mozilla-mobile:main May 12, 2022
@travis79
Copy link
Member

Data Review

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary.

  1. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the "Send Usage Data" preference in the application settings.

  1. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, collection to end in v112

  1. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical data

  1. Is the data collection request for default-on or default-off?

Default-on

  1. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

  1. Is the data collection covered by the existing Firefox privacy notice?

Yes

  1. Does the data collection use a third-party collection tool?

No

Result

data-review+

Copy link
Member

@travis79 travis79 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data-review+

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
🛬 needs landing PRs that are ready to land
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants