Krippendorff's Alpha

Krippendorff's alpha is a reliability coefficient for any number of raters that handles missing ratings and nominal, ordinal, interval or ratio data by comparing observed with expected disagreement.

Krippendorff의 알파는 평정자 수와 결측에 상관없이 명목·서열·등간·비율 자료 모두에 대해 관찰된 불일치와 기대 불일치를 비교하는 신뢰도 계수입니다.

Four annotators coded 300 social media posts, but not everyone coded every post; how reliable is the coding?

주석자 네 명이 SNS 게시물 300개를 코딩했는데 모두가 모든 게시물을 코딩하지는 않았습니다. 코딩의 신뢰도는 어떤가요?

Krippendorff's alpha equals one minus the ratio of observed disagreement within units to the disagreement expected by chance across all pairable values, with a distance function matched to the measurement level. Values of at least .800 are customary for firm conclusions and .667 for tentative ones.

When to use Krippendorff's Alpha

• Several raters code the same units and some ratings are missing
• The data are nominal, ordinal, interval or ratio and you want one coefficient family
• Content analysis or annotation studies requiring a standard reliability index

When to use something else

• Exactly two raters with complete nominal data where Cohen's kappa is conventional in the field
• Continuous measurements comparing two methods, where Bland–Altman analysis or Lin's concordance correlation is better

Assumptions and requirements

• Units are independent
• Raters code independently
• The distance metric matches the measurement level

What the analysis reports

• Alpha at the chosen measurement level
• Bootstrap 95% CI
• Probability that alpha falls below a chosen minimum
• Number of pairable values and value frequencies

How to report it

Inter-rater reliability at the [level] level across [value] units and [value] raters was Krippendorff's α = [value], 95% CI [[value], [value]].

Related methods

• Fleiss' Kappa
• Cohen's Kappa
• Intraclass Correlation (ICC)

Run Krippendorff's Alpha on Quorant