VisibilityOS by PodPitch

AI reviewer agreement calculator

Two teams can both agree 75% of the time and still have very different patterns of agreement. Count how each reviewer labeled the same answers to see the difference.

Enter jointly reviewed answers

Use answers both reviewers could read. A named brand must be the company you are checking.

Reviewer A by row and reviewer B by column
A ↓ / B →NamedAbsent
Named
Absent

Both examples are fictional. Your counts stay in this browser tab.

How often the reviewers agreed

Raw agreement across 12 answers

Expected agreement from label rates
50%
Cohen’s kappa
+0.500

Kappa compares the raw agreement with the agreement expected from how often each reviewer uses each label.

Why the same 75% can tell two stories

Both 12-answer examples have nine agreements. In the mixed-label example, the reviewers agree five times on absence and four times on a mention. Expected agreement is 50% and kappa is +0.500. In the rare-mentions example, all nine agreements are absences. Given each reviewer’s label frequency, expected agreement is 77.8% and kappa is −0.125.

Report the four counts alongside both measures so readers can see what the reviewers agreed on. Read the disputed answers and check the brand definition before using the labels to judge visibility.