Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Opting out of telemetry doesn't opt out of A/B testing if we have local experiments #6270

Closed
kimadeline opened this issue Jun 20, 2019 · 1 comment
Assignees
Labels
bug Issue identified by VS Code Team member as probable bug

Comments

@kimadeline
Copy link

A/B testing issue: #5042

Environment data

  • VS Code version: 1.35.1
  • Extension version (available under the Extensions sidebar): commit 6b1b8a3
  • OS and version: Mac (OSX 10.14.5)
  • Python version (& distribution if applicable, e.g. Anaconda): 3.7
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
  • Relevant/affected Python packages and their versions: N/A
  • Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): Jedi

Expected behaviour

Opting out of telemetry opts out of A/B testing, even if the local experiments.json is populated.

Actual behaviour

Opting out of telemetry doesn't prevent the local experiments.json file from being read and enabling experiments for the user.

Steps to reproduce:

  1. Update experiments.json with:
[
    {
        "name": "LS - control",
        "salt": "LS",
        "min": 0,
        "max": 10
    },
    {
        "name": "LS - enabled",
        "salt": "LS",
        "min": 85,
        "max": 100
    },
    {
        "name": "AlwaysDisplayTestExplorer - enabled",
        "salt": "LS",
        "min": 70,
        "max": 90
    },
    {
        "name": "AlwaysDisplayTestExplorer - control",
        "salt": "LS",
        "min": 40,
        "max": 50
    }
]
  1. Check your hash (see @karrtikr 's comment) and salt and see which experiment(s) you belong to (in my case I had LS - enabled and AlwaysDisplayTestExplorer - enabled)
  2. Go to settings and un-check 'Telemetry: Enable Telemetry' (or set telemetry.enableTelemetry to false)
  3. Set a breakpoint at line 106 of experiments.ts
  4. Start debugging the extension inside a container and notice that the breakpoint is being hit

Logs

Output for Python in the Output panel after disabling telemetry:

User belongs to experiment group 'LS - enabled'
User belongs to experiment group 'AlwaysDisplayTestExplorer - enabled'
Starting Microsoft Python language server.
@kimadeline kimadeline added bug Issue identified by VS Code Team member as probable bug triage-needed Needs assignment to the proper sub-team labels Jun 20, 2019
@karrtikr karrtikr added experimenting Feature is part of an experiment feature-* labels Jun 20, 2019
@ghost ghost removed triage-needed Needs assignment to the proper sub-team labels Jun 20, 2019
@DonJayamanne DonJayamanne added this to the 2019 - June Sprint 12 milestone Jun 20, 2019
@karrtikr karrtikr self-assigned this Jun 20, 2019
@kimadeline
Copy link
Author

Validated ✅

@ghost ghost removed experimenting Feature is part of an experiment needs PR labels Jun 24, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jul 2, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Issue identified by VS Code Team member as probable bug
Projects
None yet
Development

No branches or pull requests

3 participants