Two dependent agreement coefficients can be tested for statistical significance
What is it about?
Researchers often compute Cohen's kappa coefficient for example on two occasions using the same subjects or two overlapping groups of subjects. The first kappa may be calculated before the raters receive any training, while the second aims at quantifying the extent of agreement among raters after they have received a formal training. The fundamental research question is to determine whether the observed difference between the 2 coefficients is statistically significant. This article shows the blueprint for executing this task.
Why is it important?
Before the publication of this article, there was no known formal procedure with broad applicability that could be used for testing two agreement coefficients for statistical significance. Our procedure handles every structure of dependency that may exist in the design of the inter-rater reliability experiment.
The following have contributed to this page: KILEM GWET
In partnership with: