To measure "treatment" effects, social science researchers typically rely on nonexperimental data. In education, school and teacher effects on students are often measured through value-added models (VAMs) that are not fully understood. We propose a framework that relates to the education production function in its most flexible form and…

The present simulation study examined the statistical properties (namely, Type I error and statistical power) of various novel randomized single-case multiple-baseline designs and associated randomized-test analyses for comparing the A- to B-phase immediate abrupt outcome changes in two independent intervention conditions. It was found that with…

The reliability of a test score is usually underestimated and the deflation may be profound, 0.40 - 0.60 units of reliability or 46 - 71%. Eight root sources of the deflation are discussed and quantified by a simulation with 1,440 real-world datasets: (1) errors in the measurement modelling, (2) inefficiency in the estimator of reliability within…

Item Response Theory (IRT) models are a popular analytic method for self report data. We show how traditional IRT models can be vulnerable to specific kinds of asymmetric measurement error (AME) in self-report data, because the models spread the error to all estimates -- even those of items that do not contribute error. We quantify the impact of…

International large-scale assessments have gained much attention since the beginning of the twenty-first century, influencing education legislation in many countries. This includes Spain, where they have been used by successive governments to justify education policy change. Unfortunately, there was a problem with the PISA 2018 reading scores for…

The primary purpose of this study is to validate a student engagement measure for its use in evaluation of a learning assistant (LA) program. A series of psychometric evaluations were made for both the original scale of Higher Education Student Engagement Scale (HESES) and its adapted version designed to be used in gauging the effectiveness of…

The aim of this study is to compare the results of correlation coefficient estimation of reliability with those obtained through the Bland-Altman plot technique. The scale was first divided into two halves using three different approaches. A linear and high-level relationship was found between the scale scores obtained from the halved forms.…

The term "creative" is commonly used in everyday language and in academic discourse to discuss the nature of artistic and innovative productions. This usage inherently implies the existence of a variable of creativity that allows different creative works to be compared. The standard definition of creativity asserts that a production must…

Trend scoring constructed response items (i.e. rescoring Time A responses at Time B) gives rise to two-way data that follow a product multinomial distribution rather than the multinomial distribution that is usually assumed. Recent work has shown that the difference in sampling model can have profound negative effects on statistics usually used to…

Despite the popularity of traditional fit index cutoffs like RMSEA [less than or equal to] 0.06 and CFI [greater than or equal to] 0.95, several studies have noted issues with overgeneralizing traditional cutoffs. Computational methods have been proposed to avoid overgeneralization by deriving cutoffs specifically tailored to the characteristics…

Determining the number of factors in exploratory factor analysis (EFA) is crucial because it affects the rest of the analysis and the conclusions of the study. Researchers have developed various methods for deciding the number of factors to retain in EFA, but this remains one of the most difficult decisions in the EFA. The purpose of this study is…

This study aims to address a gap in the social and behavioral sciences literature concerning interaction effects between latent factors in multiple-group analysis. By comparing two approaches for estimating latent interactions within multiple-group analysis frameworks using simulation studies and empirical data, we assess their relative merits.…

The multidimensional forced-choice (MFC) format is an alternative to rating scales in which participants rank items according to how well the items describe them. Currently, little is known about how to detect careless responding in MFC data. The aim of this study was to adapt a number of indices used for rating scales to the MFC format and…

Evaluating scale comparability in international large-scale assessments depends on measurement invariance (MI). The root mean square deviation (RMSD) is a standard method for establishing MI in several programs, such as the Programme for International Student Assessment and the Programme for the International Assessment of Adult Competencies.…

Classroom observational protocols, in which raters observe and score the quality of teachers' instructional practices, are often used to evaluate teachers for consequential purposes despite evidence that scores from such protocols are frequently driven by factors, such as rater and temporal effects, that have little to do with teacher quality. In…