Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excluded users who marked the DP are permanently excluded #2137

Open
zackcl opened this issue Nov 26, 2024 · 1 comment
Open

Excluded users who marked the DP are permanently excluded #2137

zackcl opened this issue Nov 26, 2024 · 1 comment
Labels
bug Something isn't working Needs Refined

Comments

@zackcl
Copy link
Collaborator

zackcl commented Nov 26, 2024

Version where bug was found:
6.1.0

Describe the bug
When an experiment is started, if a user who is not in the include list (or is in the exclude list) visits the experiment's decision point and calls /init, /assign, and /mark (with a null assignedCondition), this user is permanently excluded from the experiment even if the user is later added to the include list (or removed from the exclude list).

To Reproduce

Scenario 1:

  1. Create an experiment with the following settings:
    { context: livehint-ai, site: problem-info2, target: tutorbot2, excludeIfReached: false, Include: none }
  2. Change the experiment status to "Enrolling"
  3. Make a /api/v5/init request with the following body:
{
    "id": "user2",
    "group": {
        "schoolId": [
            "school2"
       ]
    }
}
  1. Make a /api/v5/assign request with the following body:
{
    "userId": "user2",
    "context": "livehint-ai"
}
  1. Make a /api/v5/mark request with the following body:
{
    "userId": "user2",
    "data": {
        "site": "problem-info2",
        "target": "tutorbot2",
        "assignedCondition": null
    },
    "status": "no condition assigned"
}
  1. Now, update the experiment to include "All" participants
  2. Repeat 3 and 4 to verify the "user2" is still excluded (no condition from /assign)

Scenario 2:

  1. Create an experiment with the following settings:
    { context: livehint-ai, site: problem-info3, target: tutorbot3, excludeIfReached: false, Include: All }
  2. Change the experiment status to "Enrolling"
  3. Make a /api/v5/init request with the following body:
{
    "id": "user3",
    "group": {
        "schoolId": [
            "school3"
       ]
    }
}
  1. Make a /api/v5/assign request with the following body:
{
    "userId": "user3",
    "context": "livehint-ai"
}
  1. Make a /api/v5/mark request with the following body (replace the assignCondition_retrieved):
{
    "userId": "user3",
    "data": {
        "site": "problem-info3",
        "target": "tutorbot3",
        "assignedCondition": {  assignCondition_retrieved  }
    },
    "status": "condition applied"
}
  1. Now, update the experiment to exclude "user3" (Note: Do not simply empty the Include table as this will include "All" users because of the bug. Instead, either replace "All" with an "Individual" typed ID "xxx", or add "user3" to the exclude table)
  2. Repeat 3 and 4 to verify the "user3" now excluded from the experiment (no condition from /assign)
  3. Make a /api/v5/mark request with the following body:
{
    "userId": "user3",
    "data": {
        "site": "problem-info3",
        "target": "tutorbot3",
        "assignedCondition": null
    },
    "status": "no condition assigned"
}
  1. Now, update the experiment to include "All" participants
  2. Repeat 3 and 4 to verify the "user3" is still excluded (no condition from /assign)

Expected behavior
A once-excluded user should not be excluded from an experiment if later included.

@zackcl zackcl added the bug Something isn't working label Nov 26, 2024
@zackcl
Copy link
Collaborator Author

zackcl commented Dec 6, 2024

@danoswaltCL To answer your questions in the meeting, the permanent exclusion only occurs when a user marks the decision point where the experiment is enrolling but the user is not in the include list (or in the exclude list). The permanent exclusion does not happen when the user visits a decision point before an experiment is defined (Hypothesis 1), or when the experiment is inactive (Hypothesis 2).

prerequisite: Wipe out the UpGrade DB to start fresh.

Hypothesis 1: When a user visits a decision point before an experiment is defined, the user will permanently be excluded from the experiment => Result: WRONG

  1. Make sure the experiment with the following settings has not been defined yet (will be defined later):
    { context: livehint-ai, site: problem-info, target: tutorbot, excludeIfReached: false, Include: ALL }

  2. Make a /api/v5/init request with the following body:

{
    "id": "user1",
    "group": {
        "schoolId": [
            "school1"
       ]
    }
}
  1. Make a /api/v5/assign request with the following body:
{
    "userId": "user1",
    "context": "livehint-ai"
}
  1. Make a /api/v5/mark request with the following body:
{
    "userId": "user1",
    "data": {
        "site": "problem-info",
        "target": "tutorbot",
        "assignedCondition": null
    },
    "status": "no condition assigned"
}
  1. Now, define the following experiment in UpGrade:
    { context: livehint-ai, site: problem-info, target: tutorbot, excludeIfReached: false, Include: ALL }

  2. Change the experiment status to "Enrolling"

  3. Repeat 2 and 3 to verify the "user1" is included in the experiment and getting a condition (expected behavior)

Hypothesis 2: When a user visits a decision point when an experiment is defined but is inactive, the user will permanently be excluded from the experiment => Result: WRONG

  1. Create an experiment with the following settings:
    { context: livehint-ai, site: problem-info2, target: tutorbot2, excludeIfReached: false, Include: All }

  2. Make a /api/v5/init request with the following body:

{
    "id": "user2",
    "group": {
        "schoolId": [
            "school2"
       ]
    }
}
  1. Make a /api/v5/assign request with the following body:
{
    "userId": "user2",
    "context": "livehint-ai"
}
  1. Make a /api/v5/mark request with the following body:
{
    "userId": "user2",
    "data": {
        "site": "problem-info2",
        "target": "tutorbot2",
        "assignedCondition": null
    },
    "status": "no condition assigned"
}
  1. Change the experiment status to "Enrolling"

  2. Repeat 2 and 3 to verify the "user2" is included in the experiment and getting a condition (expected behavior)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Needs Refined
Projects
Status: No status
Development

No branches or pull requests

1 participant