Inter rater agreement kappa, also known as Cohen’s kappa, is a statistical measure that assesses the degree of agreement between two or more raters’ judgments. In other words, it measures how well two or more raters agree on a specific task such as coding text or the presence of a certain feature on a product.
Inter rater agreement kappa ranges from -1 to 1, with 1 indicating perfect agreement and -1 indicating complete disagreement. A score of 0 indicates chance agreement. Inter rater agreement kappa is commonly used in research and evaluation studies to measure the reliability of data collected by multiple raters.
Why is inter rater agreement kappa important?
Inter rater agreement kappa is important because it provides a quantitative assessment of the reliability of data collected by multiple raters. This information is crucial in research and evaluation studies because it ensures that the data collected is consistent and accurate.
For example, if two or more raters are coding text for a study, inter rater agreement kappa can be used to assess the degree of agreement between their coding. If their agreement is low, it may indicate that the coding process needs to be refined or that the raters need additional training to ensure consistency in their coding.
How to calculate inter rater agreement kappa
Inter rater agreement kappa can be calculated using the following formula:
Kappa = (Po – Pe) / (1 – Pe)
Where:
Po = the observed proportional agreement between the raters
Pe = the proportion of agreement expected by chance
To calculate Po, you would first tally the number of agreements between the raters and divide by the total number of ratings made. To calculate Pe, you would use the following formula:
Pe = (a1 × b1 + a2 × b2 + … + an × bn) / (N × (N – 1))
Where:
a1, a2, …, an = the marginal totals of rater 1
b1, b2, …, bn = the marginal totals of rater 2
N = the total number of ratings
Interpreting inter rater agreement kappa
As previously mentioned, inter rater agreement kappa ranges from -1 to 1. A score of 1 indicates perfect agreement, while a score of 0 indicates chance agreement. A score below 0 indicates that the raters are less consistent than would be expected by chance.
In general, a kappa score of 0.81 or higher indicates excellent agreement, while a score of 0.61-0.80 indicates substantial agreement. Scores of 0.41-0.60 indicate moderate agreement, while scores of 0.21-0.40 indicate fair agreement. Scores below 0.21 indicate poor agreement.
Conclusion
Inter rater agreement kappa is an important statistical measure in research and evaluation studies that assesses the degree of agreement between two or more raters’ judgments. It provides a quantitative assessment of the reliability of data collected by multiple raters and helps ensure that the data collected is consistent and accurate. Inter rater agreement kappa can be calculated using a formula and interpreted based on the score obtained.