Inter-Annotator Agreement › The Problem of Chance Agreement
Language AI HandbookAll chapters Inter-Annotator Agreement › The Problem of Chance AgreementOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Consider a simple binary classification task: determining whether a movie review is positive or negative. Two annotators label 100 reviews. They agree on 75 reviews and disagree on 25. Their raw agreement is 75%, which seems respectable. But what if 90 of those reviews are positive, a highly imbalanced dataset?
If both annotators simply guessed "positive" every time without reading a single review, they would agree on 90 reviews by chance alone. Their 75% observed agreement would be below chance expectation, showing systematic disagreement masked by class imbalance. Raw agreement percentages, often called "percentage of agreement" or , fail to tell us anything useful without accounting for how much agreement chance alone would produce.
This prevalence problem becomes even more acute in real-world NLP tasks. Named entity recognition datasets often have a high proportion of "O" (outside) tokens compared to entity tokens. If 95% of tokens have no entity label, two annotators who both copy-paste the majority class would agree 90% of the time. A chance-corrected coefficient would expose this false agreement immediately.
A chance-corrected agreement coefficient compares observed agreement () against expected agreement by chance (), normalized by the maximum possible improvement:
where:
- : the observed agreement proportion (the fraction of items on which raters agree)
- : the expected agreement by chance (the fraction of items on which raters would agree if guessing randomly based on marginal distributions)
When observed agreement equals chance agreement, the coefficient is zero. When observed agreement is perfect (1.0), the coefficient reaches 1.0. Negative values indicate agreement worse than chance.
The denominator represents the maximum possible improvement over chance. If chance agreement is already high (say 0.9 due to class imbalance), there is little room for improvement, making high coefficient values difficult to achieve even with careful annotators. This explains why kappa coefficients sometimes show paradoxical behavior that we examine later. Understanding this denominator is key to understanding both the power and the limitations of chance-corrected metrics.
The logic behind chance-corrected agreement can be thought of as asking: "Given how frequently each annotator uses each label, what agreement would we expect if their decisions were statistically independent?" If annotator A uses the "positive" label 70% of the time and annotator B uses it 60% of the time, and they are operating independently, we would expect them to both say "positive" on of items purely by coincidence. Any agreement beyond that reflects reliability in the annotation process.