Reliability and Consistency

Back to Index

Cronbach’s \(\alpha\)

It measures inter-correlation among items

n <- 100
dt <- data.table(
    var1 = rnorm(n)
  , noise = rnorm(n)
)
dt <- dt[, var2 := var1 * (2 + noise / 5) + noise ^ 2
         ][, var3 := var1 * (noise / 10) + var1 ^ (noise)]


library(psych)
get_cronbach <- function(data){
    a <- psych::alpha(data, warnings = FALSE)
    return(a$total$raw_alpha)
}


ttl <- expression(alpha + beta)
get_cronbach(dt) %>% prt(caption = "Cronbach's $\\alpha$")

Cronbach’s \(\alpha\)
x
-0.1357383

Cohn’s Kappa

n <- 100
dt <- data.table(
    var1 = rnorm(n)
  , noise = rnorm(n)
)
dt <- dt[, var2 := var1 * noise + noise / 10
         ][, var3 := var1 * (noise / 10) + var1 ^ (noise)]

library(psych)
cohen <- cohen.kappa(dt[, list(var1, var2)], w = NULL, alpha = .05)

print(cohen)

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels = levels)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries lower estimate upper unweighted kappa 0.0 0.0 0.0 weighted kappa -0.2 -0.2 -0.2

Number of subjects = 100

## cohen <- cohen$confid[1,]

## cohen %>% prt()

Krippendorff’s \(\alpha\)

Citation
- Matthias Gamer, Jim Lemon and Ian Fellows Puspendra Singh puspendra.pusp22@gmail.com (2019). irr: Various Coefficients of Interrater Reliability and Agreement. R package version 0.84.1. https://CRAN.R-project.org/package=irr
- Krippendorff, K. (2011). Computing Krippendorff’s Alpha-Reliability. Retrieved from https://repository.upenn.edu/asc_papers/43

n <- 50
dt <- data.table(
    var1 = factor(sample(x = 1:5, size = n, replace = TRUE), levels = 1:5, ordered = TRUE)
  , var2 = factor(sample(x = 1:5, size = n, replace = TRUE), levels = 1:5, ordered = TRUE)
  , var3 = factor(sample(x = 1:5, size = n, replace = TRUE), levels = 1:5, ordered = TRUE)
)

library(irr)

t <- data.matrix(dt)
t <- t(t)
tkp <- irr::kripp.alpha(t, method = "ordinal")

data.table(subjects = tkp$subjects, raters = tkp$raters, alpha = tkp$value) %>% prt()

subjects	raters	alpha
50	3	0.1148694

R sessionInfo

sessionInfo()

R version 4.2.0 (2022-04-22) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 20.04.3 LTS

Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale: [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] irr_0.84.1 lpSolve_5.6.15 psych_2.2.5
[4] Wu_0.0.0.9000 flexdashboard_0.5.2 lme4_1.1-29
[7] Matrix_1.4-0 mgcv_1.8-38 nlme_3.1-152
[10] png_0.1-7 scales_1.2.0 nnet_7.3-16
[13] labelled_2.9.1 kableExtra_1.3.4 plotly_4.10.0
[16] gridExtra_2.3 ggplot2_3.3.6 DT_0.23
[19] tableone_0.13.2 magrittr_2.0.3 lubridate_1.8.0
[22] dplyr_1.0.9 plyr_1.8.7 data.table_1.14.2
[25] rmdformats_1.0.4 knitr_1.39

loaded via a namespace (and not attached): [1] httr_1.4.3 sass_0.4.1 tidyr_1.2.0 jsonlite_1.8.0
[5] viridisLite_0.4.0 splines_4.2.0 bslib_0.3.1 assertthat_0.2.1 [9] highr_0.9 yaml_2.3.5 pillar_1.8.0 lattice_0.20-45
[13] glue_1.6.2 digest_0.6.29 rvest_1.0.2 minqa_1.2.4
[17] colorspace_2.0-3 htmltools_0.5.3 survey_4.1-1 pkgconfig_2.0.3
[21] haven_2.5.0 bookdown_0.27 purrr_0.3.4 webshot_0.5.3
[25] svglite_2.1.0 tibble_3.1.8 generics_0.1.3 ellipsis_0.3.2
[29] withr_2.5.0 klippy_0.0.0.9500 lazyeval_0.2.2 mnormt_2.1.0
[33] cli_3.3.0 survival_3.2-13 evaluate_0.15 fansi_1.0.3
[37] MASS_7.3-54 forcats_0.5.1 xml2_1.3.3 tools_4.2.0
[41] hms_1.1.1 mitools_2.4 lifecycle_1.0.1 stringr_1.4.0
[45] munsell_0.5.0 compiler_4.2.0 jquerylib_0.1.4 systemfonts_1.0.4 [49] rlang_1.0.4 grid_4.2.0 nloptr_2.0.3 rstudioapi_0.13
[53] htmlwidgets_1.5.4 rmarkdown_2.14 boot_1.3-28 gtable_0.3.0
[57] DBI_1.1.2 R6_2.5.1 fastmap_1.1.0 utf8_1.2.2
[61] stringi_1.7.8 parallel_4.2.0 Rcpp_1.0.9 vctrs_0.4.1
[65] tidyselect_1.1.2 xfun_0.31