This article provides a conceptual framework for considering both the theoretical and methodological factors that underpin the successful validation of a competency framework. Drawing on educational assessment literature, this article argues that a valid competency framework relates to an interpretive judgement of the credibility of the claims…

Participatory modeling (PM) is an engaged research methodology for creating analog or computer-based models of complex systems, such as socio-environmental systems. Used across a range of fields, PM centers stakeholder knowledge and participation to create more internally valid models that can inform policy and increase engagement and trust…

Evidence-based assessment (EBA) requires that investigators employ scientific theories and research findings to guide decisions about what domains to measure, how and when to measure them, and how to make decisions and interpret results. To implement EBA, investigators need high-quality assessment tools along with evidence-based processes. We…

Immersive virtual reality (VR) carries important potential, both for the creation of scientific knowledge and also for its communication. This is particularly important for studies of human spatial cognition, where psychologists now possess the power to combine the scale and fidelity of the real world with the malleability and control of the…

Purpose: Well-designed assessment methodologies and various cognitive diagnosis models (CDMs) to extract diagnostic information about examinees' individual strengths and weaknesses have been developed. Due to this novelty, as well as educational specialists' lack of familiarity with CDMs, their applications are not widespread. This article aims at…

Cognitive diagnosis models (CDMs) allow for the extraction of fine-grained, multidimensional diagnostic information from appropriately designed tests. In recent years, interest in such models has grown as formative assessment grows in popularity. Many dichotomous as well as several polytomous CDMs have been proposed in the last two decades, but…

Drawing upon his experience, more than 60 years ago, as a psychometric support person to a very special teacher of brain damaged children, the author of this article reflects on the productive use of educational assessments and data from them to educate - assessment in the service of learning. Findings from the Gordon Commission on the Future of…

In causal matching designs, some control subjects are often left unmatched, and some covariates are often left unmodeled. This article introduces "rebar," a method using high-dimensional modeling to incorporate these commonly discarded data without sacrificing the integrity of the matching design. After constructing a match, a researcher…

Data analysis is a significant methodological component when conducting quantitative education studies. Guidelines for conducting data analyses in quantitative education studies are common but often underemphasize four important methodological components impacting the validity of inferences: quality of constructed measures, proper handling of…

With the widespread use of learning analytics (LA) tools, there is a need to explore how these technologies can be used to enhance teaching and learning. Little research has been conducted on what human processes are necessary to facilitate meaningful adoption of LA. The research problem is that there is a lack of evidence-based guidance on how…

This paper addresses a relatively unexplored area in the field of learning analytics: how analytics are taken up and used as part of teaching and learning processes. Initial steps are taken towards developing design knowledge for this "middle space," with a focus on students as analytics users. First, a core set of challenges for…

Psychological testing can serve as a prototype on which to base a data-generated approach to instructional design. In "testing batteries" tests are used to predict achievement. In the proposed approach batteries of prescriptions would be used to produce achievement. In creating "test batteries" tests are selected for their…

This study proposes a structured constructs model (SCM) to examine measurement in the context of a multidimensional learning progression (LP). The LP is assumed to have features that go beyond a typical multidimentional IRT model, in that there are hypothesized to be certain cross-dimensional linkages that correspond to requirements between the…

Information and communication technology (ICT)-enhanced research methods such as educational data mining (EDM) have allowed researchers to effectively model a broad range of constructs pertaining to the student, moving from traditional assessments of knowledge to assessment of engagement, meta-cognition, strategy and affect. The automated…

Quantitative research on sexual minority youths (SMYs) has likely contributed to misperceptions about the risk and deviance of this population. In part because it often relies on self-reported data from population-based self-administered questionnaires, this research is susceptible to misclassification bias whereby youths who are not SMYs are…