Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add globally accessible A/B testing opt-out setting #36295

Open
tmancey opened this issue Feb 22, 2024 · 7 comments
Open

Add globally accessible A/B testing opt-out setting #36295

tmancey opened this issue Feb 22, 2024 · 7 comments
Labels
needs-discussion Although the issue is clear, we haven't yet reached a decision about the right solution. OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality

Comments

@tmancey
Copy link
Contributor

tmancey commented Feb 22, 2024

Preferably under the Privacy and Security setting, along with P3A and the ping setting

@tmancey tmancey added OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality labels Feb 22, 2024
@tmancey tmancey changed the title Add globally accessible A/B testing opt-out Add globally accessible A/B testing opt-out setting Feb 22, 2024
@ShivanKaul
Copy link
Collaborator

Given that almost all A/B tests are going to be via P3A, I don't think it makes sense to have a separate toggle for this.

@haddadi
Copy link

haddadi commented Feb 22, 2024

most A/B product/design tests might not have any user data (will a red lion logo be more effective on shields than a blue lion logo?) in which case, there is no user data involved.

@diracdeltas
Copy link
Member

diracdeltas commented Feb 22, 2024

I noted this in a DM to Hamed, but A/B test data (griffin state) is sent in crash reports too not just P3A. I think we are all on the same page that it'd be good to have a global opt out for A/B testing, so my understanding is that we can do this 3 ways:

  1. Tie it to P3A opt out, and P3A opt out also opts out of sending Griffin data in crash reports. This seems a bit non-intuitive to me.
  2. Explain in settings that Brave does A/B testing and to truly opt out from A/B test data collection you have to disable both the P3A and crash report flags. EDIT: see comment below, this is a lot clunkier than I initially thought.
  3. Separate toggle for A/B testing. This would be cleanest. To avoid cluttering privacy settings, we can create a new section in brave://settings/privacy (which is really privacy+security) called "Your data" and put p3a, usage pings, crash reports, a/b test in there.

EDIT: Noting that https://github.com/brave/reviews/issues/1507 also collects A/B experiment data outside of P3A. Given that we'd have to make sure if opt out is tied to P3A, it also disables this data collection in ads confirmations. Or explain that to truly opt out of A/B testing you have to also opt out of rewards/news.

@ShivanKaul
Copy link
Collaborator

FWIW I just don't think we need to treat A/B tests as special cases of data collection.

@tmancey tmancey added the needs-discussion Although the issue is clear, we haven't yet reached a decision about the right solution. label Feb 23, 2024
@tmancey
Copy link
Contributor Author

tmancey commented Feb 23, 2024

Added needs-discussion label. Thanks

@diracdeltas
Copy link
Member

Is this actually a duplicate of #15711 ? It seems in there we had consensus to add a new toggle for opting out of "staged roll-outs, feature testing and user studies" which is even more comprehensive than this issue, which is just to opt out of user studies without actually removing the ability to get new griffin variations.

@ShivanKaul
Copy link
Collaborator

Yeah, that one is a full-on "no griffin for me" toggle. If we add a Data Collection subsection like is the plan in #36363, it would be fine to add a Griffin opt-out toggle there, and that should cover this case as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-discussion Although the issue is clear, we haven't yet reached a decision about the right solution. OS/Android Fixes related to Android browser functionality OS/Desktop OS/iOS Fixes related to iOS browser functionality
Projects
None yet
Development

No branches or pull requests

4 participants