The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305D210045 and R305D240032 to the University of Kansas Center for Research, Inc., ATLAS. The opinions expressed are those of the authors and do not represent the views of the the Institute or the U.S. Department of Education.
What are diagnostic models?
Traditional assessments and psychometric models measure an overall skill or ability
Assume a continuous latent trait
Traditional measurement
The output is a weak ordering of eras due to error in estimates
Confident Taylor Swift (debut) is the worst
Not confident on ordering toward the middle of the distribution
Limited in the types of questions that can be answered.
Why is Taylor Swift (debut) so low?
What aspects do each era demonstrate proficiency or competency of?
How much skill is “enough” to be competent?
Diagnostic measurement
Designed to be multidimensional
No continuum of student achievement
Categorical constructs
Usually binary (e.g., master/nonmaster, proficient/not proficient)
Several different names in the literature
Diagnostic classification models (DCMs)
Cognitive diagnostic models (CDMs)
Skills assessment models
Latent response models
Restricted latent class models
Diagnostic music assessment
Rather than measuring overall musical knowledge, we can break music down into set of skills or attributes
Songwriting
Production
Vocals
Attributes are categorical, often dichotomous (e.g., proficient vs. non-proficient)
Diagnostic classification models
DCMs place individuals into groups according to proficiency of multiple attributes
songwriting
production
vocals
Benefits of DCMs
Fine-grained, multidimensional results. Answer more questions:
Why is Taylor Swift (debut) so low?
Subpar songwriting, production, and vocals
What aspects are albums competent/proficient in?
DCMs provide classifications directly
High reliability with fewer items
Less information need to classify than to place precisely along a scale
Using DCMs to improve student outcomes
Improved software for diagnostic models
measr: R package for Bayesian psychometric measurement using Stan
Easily specify and estimate a DCM
Wide variety of DCMs (e.g., LCDM, DINA, C-RUM)
Defined attribute relationships and dependencies
Supports maximum likelihood and full MCMC model estimation
Powerful model evaluation tools
Model fit using posterior predictive model checks
Model comparisons with leave-one-out cross validation
