Cohen's Kappa and Weighted Kappa
Computes the agreement rates Cohen's kappa and weighted kappa and their confidence intervals.
CohenKappa(x, y = NULL, weights = c("Unweighted", "Equal-Spacing", "Fleiss-Cohen"),
conf.level = NA, ...)x |
can either be a numeric vector or a confusion matrix. In the latter case x must be a square matrix. |
y |
NULL (default) or a vector with compatible dimensions to |
weights |
either one out of |
conf.level |
confidence level of the interval. If set to |
... |
further arguments are passed to the function |
Cohen's kappa is the diagonal sum of the (possibly weighted) relative frequencies, corrected for expected values and standardized by its maximum value.
The equal-spacing weights (see Cicchetti and Allison 1971) are defined by
1 - \frac{|i - j|}{r - 1}
r being the number of columns/rows, and the Fleiss-Cohen weights by
1 - \frac{(i - j)^2}{(r - 1)^2}
The latter attaches greater importance to closer disagreements.
Data can be passed to the function either as matrix or data.frame in x, or as two numeric vectors x and y. In the latter case table(x, y, ...) is calculated. Thus NAs are handled the same way as table does. Note that tables are by default calculated without NAs. The specific argument useNA can be passed via the ... argument.
The vector interface (x, y) is only supported for the calculation of unweighted kappa. For 2 factors with different levels we cannot ensure a reproducible construction of a confusion table, which is independent of the order of x and y. All weights might lead to inconsistent results.
Thus the function will raise an error in such cases.
if no confidence intervals are requested:
the estimate as numeric value
else a named numeric vector with 3 elements
kappa |
estimate |
lwr.ci |
lower confidence interval |
upr.ci |
upper confidence interval |
David Meyer <david.meyer@r-project.org>, some slight changes Andri Signorell <andri@signorell.net>
Cohen, J. (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37-46.
Everitt, B.S. (1968), Moments of statistics kappa and weighted kappa. The British Journal of Mathematical and Statistical Psychology, 21, 97-103.
Fleiss, J.L., Cohen, J., and Everitt, B.S. (1969), Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, 72, 332-327.
Cicchetti, D.V., Allison, T. (1971) A New Procedure for Assessing Reliability of Scoring EEG Sleep Recordings American Journal of EEG Technology, 11, 101-109.
# from Bortz et. al (1990) Verteilungsfreie Methoden in der Biostatistik, Springer, pp. 459
m <- matrix(c(53, 5, 2,
11, 14, 5,
1, 6, 3), nrow=3, byrow=TRUE,
dimnames = list(rater1 = c("V","N","P"), rater2 = c("V","N","P")) )
# confusion matrix interface
CohenKappa(m, weight="Unweighted")
# vector interface
x <- Untable(m)
CohenKappa(x$rater1, x$rater2, weight="Unweighted")
# pairwise Kappa
rating <- data.frame(
rtr1 = c(4,2,2,5,2, 1,3,1,1,5, 1,1,2,1,2, 3,1,1,2,1, 5,2,2,1,1, 2,1,2,1,5),
rtr2 = c(4,2,3,5,2, 1,3,1,1,5, 4,2,2,4,2, 3,1,1,2,3, 5,4,2,1,4, 2,1,2,3,5),
rtr3 = c(4,2,3,5,2, 3,3,3,4,5, 4,4,2,4,4, 3,1,1,4,3, 5,4,4,4,4, 2,1,4,3,5),
rtr4 = c(4,5,3,5,4, 3,3,3,4,5, 4,4,3,4,4, 3,4,1,4,5, 5,4,5,4,4, 2,1,4,3,5),
rtr5 = c(4,5,3,5,4, 3,5,3,4,5, 4,4,3,4,4, 3,5,1,4,5, 5,4,5,4,4, 2,5,4,3,5),
rtr6 = c(4,5,5,5,4, 3,5,4,4,5, 4,4,3,4,5, 5,5,2,4,5, 5,4,5,4,5, 4,5,4,3,5)
)
PairApply(rating, FUN=CohenKappa, symmetric=TRUE)
# Weighted Kappa
cats <- c("<10%", "11-20%", "21-30%", "31-40%", "41-50%", ">50%")
m <- matrix(c(5,8,1,2,4,2, 3,5,3,5,5,0, 1,2,6,11,2,1,
0,1,5,4,3,3, 0,0,1,2,5,2, 0,0,1,2,1,4), nrow=6, byrow=TRUE,
dimnames = list(rater1 = cats, rater2 = cats) )
CohenKappa(m, weight="Equal-Spacing")
# supply an explicit weight matrix
ncol(m)
(wm <- outer(1:ncol(m), 1:ncol(m), function(x, y) {
1 - ((abs(x-y)) / (ncol(m)-1)) } ))
CohenKappa(m, weight=wm, conf.level=0.95)
# however, Fleiss, Cohen and Everitt weight similarities
fleiss <- matrix(c(
106, 10, 4,
22, 28, 10,
2, 12, 6
), ncol=3, byrow=TRUE)
#Fleiss weights the similarities
weights <- matrix(c(
1.0000, 0.0000, 0.4444,
0.0000, 1.0000, 0.6666,
0.4444, 0.6666, 1.0000
), ncol=3)
CohenKappa(fleiss, weights)Please choose more modern alternatives, such as Google Chrome or Mozilla Firefox.