Monte Carlo simulation studies of reliability in psychometrics

A methodological review

Abstract

Monte Carlo simulation studies are widely used in reliability research. This study reviewed 85 published Monte Carlo simulation studies investigating reliability. The review focused on the prevalence of particular reliability estimation methods and estimators, as well as adherence to previous recommendations for the Monte Carlo simulation method. It appears researchers do not fully adhere to these recommendations. Most of the reviewed studies have limitations in at least one of the following: reporting on the data generation procedure, selection of the number of replications, selection of conditions, benchmark utilization, and performance evaluation. Findings also suggest internal consistency in general and coefficient α are the most prevalent. Conversely, some reliability estimation methods and estimators that can be useful under many empirical conditions appear to be mostly overlooked. In the case of internal consistency, these are relatively obscure forms of α, λ₂, λ₄, μ-series, Kristof's coefficient, Feldt-Gilmer coefficient, maximal reliability, greatest lower bound to reliability, ω family, structural equation modeling-based coefficients, and internal consistency confidence intervals. Overlooked reliability estimation methods are parallel forms, test-retest, multilevel reliability, latent class-based reliability, reliability of an individual, and Bayesian reliability. Suggestions for future research have been offered.