You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
When an experiment is started, if a user who is not in the include list (or is in the exclude list) visits the experiment's decision point and calls /init, /assign, and /mark (with a null assignedCondition), this user is permanently excluded from the experiment even if the user is later added to the include list (or removed from the exclude list).
To Reproduce
Scenario 1:
Create an experiment with the following settings:
{ context: livehint-ai, site: problem-info2, target: tutorbot2, excludeIfReached: false, Include: none }
Change the experiment status to "Enrolling"
Make a /api/v5/init request with the following body:
Now, update the experiment to include "All" participants
Repeat 3 and 4 to verify the "user2" is still excluded (no condition from /assign)
Scenario 2:
Create an experiment with the following settings:
{ context: livehint-ai, site: problem-info3, target: tutorbot3, excludeIfReached: false, Include: All }
Change the experiment status to "Enrolling"
Make a /api/v5/init request with the following body:
Now, update the experiment to exclude "user3" (Note: Do not simply empty the Include table as this will include "All" users because of the bug. Instead, either replace "All" with an "Individual" typed ID "xxx", or add "user3" to the exclude table)
Repeat 3 and 4 to verify the "user3" now excluded from the experiment (no condition from /assign)
Make a /api/v5/mark request with the following body:
@danoswaltCL To answer your questions in the meeting, the permanent exclusion only occurs when a user marks the decision point where the experiment is enrolling but the user is not in the include list (or in the exclude list). The permanent exclusion does not happen when the user visits a decision point before an experiment is defined (Hypothesis 1), or when the experiment is inactive (Hypothesis 2).
prerequisite: Wipe out the UpGrade DB to start fresh.
Hypothesis 1: When a user visits a decision point before an experiment is defined, the user will permanently be excluded from the experiment => Result: WRONG
Make sure the experiment with the following settings has not been defined yet (will be defined later):
{ context: livehint-ai, site: problem-info, target: tutorbot, excludeIfReached: false, Include: ALL }
Make a /api/v5/init request with the following body:
Now, define the following experiment in UpGrade:
{ context: livehint-ai, site: problem-info, target: tutorbot, excludeIfReached: false, Include: ALL }
Change the experiment status to "Enrolling"
Repeat 2 and 3 to verify the "user1" is included in the experiment and getting a condition (expected behavior)
Hypothesis 2: When a user visits a decision point when an experiment is defined but is inactive, the user will permanently be excluded from the experiment => Result: WRONG
Create an experiment with the following settings:
{ context: livehint-ai, site: problem-info2, target: tutorbot2, excludeIfReached: false, Include: All }
Make a /api/v5/init request with the following body:
Version where bug was found:
6.1.0
Describe the bug
When an experiment is started, if a user who is not in the include list (or is in the exclude list) visits the experiment's decision point and calls
/init
,/assign
, and/mark
(with anull
assignedCondition), this user is permanently excluded from the experiment even if the user is later added to the include list (or removed from the exclude list).To Reproduce
Scenario 1:
{ context:
livehint-ai
, site:problem-info2
, target:tutorbot2
, excludeIfReached:false
, Include:none
}/api/v5/init
request with the following body:/api/v5/assign
request with the following body:/api/v5/mark
request with the following body:/assign
)Scenario 2:
{ context:
livehint-ai
, site:problem-info3
, target:tutorbot3
, excludeIfReached:false
, Include:All
}/api/v5/init
request with the following body:/api/v5/assign
request with the following body:/api/v5/mark
request with the following body (replace theassignCondition_retrieved
):/assign
)/api/v5/mark
request with the following body:/assign
)Expected behavior
A once-excluded user should not be excluded from an experiment if later included.
The text was updated successfully, but these errors were encountered: