For #5312 - Add Telemetry for Tracking Protection #5414

ekager · 2019-09-18T20:03:22Z

Pull Request checklist

Quality: This PR builds and passes detekt/ktlint checks (A pre-push hook is recommended)
Tests: This PR includes thorough tests or an explanation of why it does not
Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features

After merge

Milestone: Make sure issues finished by this pull request are added to the milestone of the version currently in development.

To download an APK when reviewing a PR:

click on Show All Checks,
click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
click on the "Fenix - assemble" task, then click "Run Artifacts".
the APK links should be on the left side of the screen, named for each CPU architecture

app/src/main/java/org/mozilla/fenix/components/metrics/Metrics.kt

ekager · 2019-09-18T20:12:49Z

Request for data collection review form

All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.

What questions will you answer with this data?

How often users interact with tracking protection panel
How often users have to add a site exception because of site problems
How often users edit tracking protection settings

Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

Usefulness and discoverability of tracking protection panel
How often users change the tracking protection settings to a lower level of protection

What alternative methods did you consider to answer these questions? Why were they not sufficient?

We currently record if TP is on/off but we have more granular options now and more UI surrounding TP

Can current instrumentation answer these questions?

Currently there are no events to track this.

List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data c ollection categories on the found on the Mozilla wiki.

All data is Category 2.

How long will this data be collected?

Until 03/01/2020

What populations will you measure?

All release, beta, and nightly users with telemetry enabled.

Please provide a general description of how you will analyze this data.

Glean / Amplitude

Where do you intend to share the results of your analysis?

Only on Glean, Amplitude and with mobile teams.

codecov-io · 2019-09-18T20:31:41Z

Codecov Report

Merging #5414 into master will decrease coverage by 0.04%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master    #5414      +/-   ##
============================================
- Coverage     13.09%   13.05%   -0.05%     
  Complexity      289      289              
============================================
  Files           252      252              
  Lines         10367    10403      +36     
  Branches       1504     1512       +8     
============================================
  Hits           1358     1358              
- Misses         8916     8952      +36     
  Partials         93       93

Impacted Files	Coverage Δ	Complexity Δ
...la/fenix/components/metrics/GleanMetricsService.kt	`0.74% <0%> (-0.04%)`	`1 <0> (ø)`
...a/org/mozilla/fenix/browser/BaseBrowserFragment.kt	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
...va/org/mozilla/fenix/components/metrics/Metrics.kt	`25.12% <0%> (-0.9%)`	`0 <0> (ø)`
...a/org/mozilla/fenix/exceptions/ExceptionDomains.kt	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
.../trackingprotection/TrackingProtectionPanelView.kt	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
...ava/org/mozilla/fenix/settings/SettingsFragment.kt	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
...zilla/fenix/settings/TrackingProtectionFragment.kt	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️
...rotection/TrackingProtectionPanelDialogFragment.kt	`0% <0%> (ø)`	`0 <0> (ø)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c866ae1...002fdac. Read the comment docs.

sblatz

Looks good, one nit 😄

docs/metrics.md

sblatz · 2019-09-20T14:01:30Z

This just has a do not land until @boek can do a data review :)

sblatz · 2019-09-20T16:05:35Z

@ekager did we ever hear back from @vesta0 about adding tracking protection setting (standard/off/strict) to the "core ping"? I feel like it's actually more useful there.

ekager · 2019-09-20T18:08:30Z

I think we want to specifically know when users feel they need to change the default TP settings and how often they do that?

liuche

A few nits - data-review+ on addressing them

Data Review Form (to be filled by Data Stewards)

Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, metrics.yaml generated docs in metrics.md

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, Fenix data controls

If the request is for permanent data collection, is there someone who will monitor the data over time?

Expires 3/2020

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 2, user interactions with and selection of different tracking protection settings

Is the data collection request for default-on or default-off?

Default on

Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No

Is the data collection covered by the existing Firefox privacy notice?

yes

Does there need to be a check-in in the future to determine whether to renew the data?

No, has expiry

Does the data collection use a third-party collection tool?

No

liuche · 2019-09-20T23:50:52Z

app/metrics.yaml

+  exception_added:
+    type: event
+    description: >
+      A user added a TP exception through the TP toggle in the panel.


Nit: can we spell out TP, since this is for docs, and TP is a less common abbreviation

liuche · 2019-09-20T23:51:40Z

app/metrics.yaml

+  panel_settings:
+    type: event
+    description: >
+      A user opened ETP settings from the panel.


Same nit: ETP is more common, but might as well spell it out too "Enhanced Tracking Protection (ETP)"

liuche · 2019-09-20T23:51:58Z

app/metrics.yaml

+  etp_shield:
+    type: event
+    description: >
+      A user pressed the TP shield icon in toolbar.


liuche · 2019-09-20T23:53:00Z

app/metrics.yaml

+    notification_emails:
+      - [email protected]
+    expires: "2020-03-01"
+  etp_shield:


I'm on the fence whether we should rename this telemetry probe to enhanced_tracking_protection_toolbar_shield

liuche · 2019-09-20T23:54:28Z

app/metrics.yaml

+  etp_tracker_list:
+    type: event
+    description: >
+      A user pressed into tracker list display in panel.


"entered tracking protection settings from toolbar panel"?

liuche · 2019-09-21T00:07:07Z

Make sure to regenerate md docs after changes too!

ekager requested review from a team as code owners September 18, 2019 20:03

Amejia481 reviewed Sep 18, 2019

View reviewed changes

app/src/main/java/org/mozilla/fenix/components/metrics/Metrics.kt Show resolved Hide resolved

ekager force-pushed the 5312-telemetry-etp branch from 95b675a to 61c6b48 Compare September 18, 2019 20:17

ekager requested a review from boek September 18, 2019 20:20

sblatz approved these changes Sep 19, 2019

View reviewed changes

docs/metrics.md Outdated Show resolved Hide resolved

sblatz added needs:data-review PR is awaiting a data review pr:do-not-land labels Sep 19, 2019

ekager force-pushed the 5312-telemetry-etp branch from 61c6b48 to b3c827a Compare September 19, 2019 20:26

ekager force-pushed the 5312-telemetry-etp branch 2 times, most recently from 1ad40df to a696015 Compare September 20, 2019 23:38

liuche approved these changes Sep 21, 2019

View reviewed changes

liuche removed the request for review from boek September 21, 2019 00:05

liuche added pr:needs-changes PRs that need some changes/fixes before they can land and removed pr:do-not-land needs:data-review PR is awaiting a data review labels Sep 21, 2019

For mozilla-mobile#5312 - Add Telemetry for Tracking Protection

002fdac

ekager force-pushed the 5312-telemetry-etp branch from a696015 to 002fdac Compare September 21, 2019 04:11

ekager removed the pr:needs-changes PRs that need some changes/fixes before they can land label Sep 23, 2019

ekager merged commit 9d9ff95 into mozilla-mobile:master Sep 23, 2019

ekager deleted the 5312-telemetry-etp branch September 23, 2019 15:29

liuche mentioned this pull request Oct 6, 2020

Issue #14142 - Telemetry renewal to 08-01-2021 #15713

Merged

3 tasks

eliserichards mentioned this pull request Jul 26, 2021

For #20516: Renew product telemetry probes expiring in August #20517

Merged

3 tasks

eliserichards mentioned this pull request Jan 28, 2022

For #22870: Renew/remove metrics set to expire in February #23453

Merged

6 tasks

MatthewTighe mentioned this pull request Jan 11, 2023

For #28190: renew telemetry expiring in 111 #28502

Merged

4 tasks

rocketsroger mentioned this pull request Oct 11, 2023

Bug 1858440 - Renew all telemetry to never expires mozilla-mobile/firefox-android#4039

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

For #5312 - Add Telemetry for Tracking Protection #5414

For #5312 - Add Telemetry for Tracking Protection #5414

ekager commented Sep 18, 2019

ekager commented Sep 18, 2019

codecov-io commented Sep 18, 2019 •

edited

Loading

sblatz left a comment

sblatz commented Sep 20, 2019

sblatz commented Sep 20, 2019 •

edited

Loading

ekager commented Sep 20, 2019

liuche left a comment

liuche Sep 20, 2019

liuche Sep 20, 2019

liuche Sep 20, 2019

liuche Sep 20, 2019

liuche Sep 20, 2019

liuche commented Sep 21, 2019

For #5312 - Add Telemetry for Tracking Protection #5414

For #5312 - Add Telemetry for Tracking Protection #5414

Conversation

ekager commented Sep 18, 2019

Pull Request checklist

After merge

To download an APK when reviewing a PR:

ekager commented Sep 18, 2019

Request for data collection review form

codecov-io commented Sep 18, 2019 • edited Loading

Codecov Report

sblatz left a comment

Choose a reason for hiding this comment

sblatz commented Sep 20, 2019

sblatz commented Sep 20, 2019 • edited Loading

ekager commented Sep 20, 2019

liuche left a comment

Choose a reason for hiding this comment

Data Review Form (to be filled by Data Stewards)

liuche Sep 20, 2019

Choose a reason for hiding this comment

liuche Sep 20, 2019

Choose a reason for hiding this comment

liuche Sep 20, 2019

Choose a reason for hiding this comment

liuche Sep 20, 2019

Choose a reason for hiding this comment

liuche Sep 20, 2019

Choose a reason for hiding this comment

liuche commented Sep 21, 2019

codecov-io commented Sep 18, 2019 •

edited

Loading

sblatz commented Sep 20, 2019 •

edited

Loading