Cohen's Kappa

Measure inter-rater reliability for categorical ratings — accounting for agreement expected by chance alone.

Cohen's Kappa (κ) is the standard measure of inter-rater reliability for categorical data. It corrects the observed proportion of agreement for the agreement expected by chance alone, making it more informative than simple percent agreement. κ = (P_observed − P_expected) / (1 − P_expected). Values of .61–.80 are considered substantial agreement; >.80 is almost perfect agreement.

Weighted Kappa allows partial credit for disagreements that are 'close' on an ordinal scale — for instance, one rater coding 'mild' and another coding 'moderate' is a smaller error than coding 'mild' vs. 'severe'. StatRyx supports both unweighted and linearly weighted kappa.

Cohen's kappa measures agreement between two raters on a categorical coding, corrected for the agreement expected by chance. κ = 0 means agreement no better than chance, 1 means perfect agreement. Percentage agreement alone overstates reliability when one category is very common, because raters agree often just by guessing that category. For ordered categories, weighted kappa gives partial credit to near misses.

When to use the Cohen's Kappa

• You have two raters independently coding the same set of observations into categorical categories
• You need to report inter-rater reliability for a qualitative coding scheme
• Your categories are ordinal and you want to give partial credit for near-agreements (weighted kappa)
• Two coders categorised interview excerpts or open survey answers and you need inter-rater reliability
• Two clinicians assigned diagnoses to the same patients

When not to use it

• You have more than two raters — use Fleiss's Kappa instead
• Your outcome is continuous — use Intraclass Correlation Coefficient (ICC) instead
• One category is extremely common (e.g., 95% 'no') — use Report percentage agreement and kappa together, or Gwet's AC1, because kappa can be low despite high agreement instead

Worked example

How to report it (APA 7)

Inter-rater reliability for the diagnostic coding scheme was assessed using Cohen's Kappa. Agreement was substantial, κ = .74 (95% CI [.64, .84]), p < .001, based on 120 coded transcripts.

Frequently asked questions

Is Cohen's Kappa free on StatRyx?

Yes. You can run it without an account (3 free analyses), and a free account adds 10 more — a one-time allowance, no payment needed. Every result includes the APA 7 write-up. Paid plans (Student from $9/month) add more analyses and Word, PDF and Excel export.

What κ value is considered acceptable?

Landis & Koch (1977) benchmarks: κ < .20 slight, .21–.40 fair, .41–.60 moderate, .61–.80 substantial, >.80 almost perfect. Many journals require κ ≥ .70 for categorical coding.

How do I report Cohen's kappa in APA 7?

Report κ with its 95% CI, the number of units coded, and the interpretation band: "Two raters coded 120 responses; agreement was substantial, κ = .74, 95% CI [.64, .84]." Mention which benchmark you used (e.g., Landis & Koch, 1977).

What is a good kappa value?

Landis and Koch (1977): .21–.40 fair, .41–.60 moderate, .61–.80 substantial, above .80 almost perfect. Many fields expect at least .70 for a coding scheme to be considered reliable; McHugh (2012) argues for stricter thresholds in health research.

Can I use Cohen's kappa with three raters?

No — Cohen's kappa is for exactly two raters. With three or more raters use Fleiss' kappa, or average the pairwise Cohen's kappas and report the range.

Run a Cohen's Kappa on your data →

Related analyses

• Chi-Square Test
• McNemar Test
• Logistic Regression