Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle IRR assignments #370

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Handle IRR assignments #370

wants to merge 1 commit into from

Conversation

AstridKery
Copy link
Contributor

Right now if the number of IRR labels needed for each data is 2 and three coders are online, the item will get passed to all three of them. This update limits that so that an item isn't passed out to that third coder if the number of coders required for each data item is 2.

It also adds back in a message to display when no cards are returned to the front end as requested by users.

@AstridKery AstridKery self-assigned this Nov 25, 2024
Copy link
Contributor

@andykawabata andykawabata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to running the pytests, I manually tested this pull request by simulating an IRR project with three coders. I tried this with 50% and 100% IRR projects, each with a two-coder limit. I was mainly testing for three things:

  1. No datum should be assigned to more than two coders at a time under any circumstances.
  2. The new restriction on data assignment doesn't prevent data that should be assigned from getting assigned.
  3. IRR still works as expected.

I had all three coders enter the annotation page before assigning any labels and then tried various combinations of labeling, skipping, and adjudicating while cross-checking the database tables to make sure everything was in the expected state.

As far as I could tell, everything is working as intended, as I was unable to find any circumstance where a datum could be labeled by more than two coders.

Note on Pytests

I did notice a limitation in tests irr_test.py where after two agreeing labels, we should be able to assert that datum.irr_ind is False. However, it does not actually get changed to False since we are only calling label_data() and not everything else that gets called when actually annotating.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants