Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better behaviour when expected disagreement is 0 for Krippendorff's alpha #19

Closed
logological opened this issue Sep 8, 2015 · 2 comments

Comments

@logological
Copy link
Member

Some measures of interannotator agreement, such as Cohen's kappa and Krippendorff's alpha, are not meaningful for cases where all annotators assigned the same label. In this case the expected disagreement is 0 and, were the measure's usual formula to be calculated, it would produce a division by 0.

When KrippendorffAlphaAgreement is told to calculate Krippendorff's alpha for such cases, calculateAgreement() returns 0.0. I think this is misleading—a value of 0.0 means that the agreement was successfully calculated, and that it was indistinguishable to a random labelling. I think that calculateAgreement() should have instead signalled to the user that it could not return a meaningful result with the data it was given. CohenKappaAgreement correctly handles this case by throwing an exception of type InsufficientDataException—maybe KrippendorffAlphaAgreement should do likewise…?

@logological
Copy link
Member Author

Hmm… I see now that CohenKappaAgreement is throwing the exception via CodingAgreementMeasure. So the problematic behaviour described above is probably a consequence of KrippendorffAlphaAgreement not implementing ICodingAgreementMeasure, as mentioned in Issue #17.

@logological
Copy link
Member Author

Looking at other IAA tools such as ReCal seems to confirm that Krippendorff's alpha is undefined when the expected disagreement is 0. I will push a change that checks getCategoryCount() and throws an exception if it's not 2 or more.

logological added a commit that referenced this issue Sep 8, 2015
KrippendorffAlphaAgreement now throws InsufficientDataException
on encountering invariant values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant