Fleiss' Kappa

Fleiss' kappa measures chance-corrected agreement when a fixed number of raters assign each subject to one of several nominal categories, and the raters need not be the same across subjects.

Fleiss 카파는 정해진 수의 평정자가 각 대상을 여러 명목 범주 중 하나로 분류할 때 우연 일치를 보정한 전체 일치도이며, 대상마다 평정자가 달라도 됩니다.

Five reviewers each classified 60 abstracts as include, exclude or unsure; what is their overall agreement?

심사자 다섯 명이 초록 60편을 포함·제외·보류로 각각 분류했는데, 전체 일치도는 어느 정도인가요?

Fleiss' kappa compares the average proportion of agreeing rater pairs per subject with the agreement expected from the overall category proportions. Category-specific kappas show which categories are judged consistently and which are confused.

When to use Fleiss' Kappa

• Three or more raters assign nominal categories
• Each subject receives the same number of ratings
• Raters may differ from subject to subject

When to use something else

• Some ratings are missing or the number of raters varies, where Krippendorff's alpha handles missing data
• Categories are ordered, where Krippendorff's alpha at the ordinal level or an intraclass correlation is better

Assumptions and requirements

• Every subject has the same number of ratings
• Ratings of different subjects are independent
• Categories are mutually exclusive and nominal

What the analysis reports

• Overall kappa with bootstrap 95% CI
• z test of kappa = 0 using the null standard error
• Category-specific kappas
• Observed and expected agreement

How to report it

Agreement among the [value] raters across [value] subjects was Fleiss' κ = [value], 95% CI [[value], [value]], z = [value], p = [value].

Related methods

• Cohen's Kappa
• Krippendorff's Alpha

Run Fleiss' Kappa on Quorant