AI reviewer agreement calculator
Two teams can both agree 75% of the time and still have very different patterns of agreement. Count how each reviewer labeled the same answers to see the difference.
Enter jointly reviewed answers
Use answers both reviewers could read. A named brand must be the company you are checking.
Both examples are fictional. Your counts stay in this browser tab.
How often the reviewers agreed
75%
Raw agreement across 12 answers
- Expected agreement from label rates
- 50%
- Cohen’s kappa
- +0.500
Kappa compares the raw agreement with the agreement expected from how often each reviewer uses each label.
Why the same 75% can tell two stories
Both 12-answer examples have nine agreements. In the mixed-label example, the reviewers agree five times on absence and four times on a mention. Expected agreement is 50% and kappa is +0.500. In the rare-mentions example, all nine agreements are absences. Given each reviewer’s label frequency, expected agreement is 77.8% and kappa is −0.125.
Report the four counts alongside both measures so readers can see what the reviewers agreed on. Read the disputed answers and check the brand definition before using the labels to judge visibility.