Model evaluation -- the process of making inferences about the performance of predictive models -- is a critical component of predictive modelling research in learning analytics. We survey the state of the practice with respect to model evaluation in learning analytics, which overwhelmingly uses only naïve methods for model evaluation or…

Testlet effects can be taken into account by incorporating specific dimensions in addition to the general dimension into the item response theory model. Three such multidimensional models are described: the bi-factor model, the testlet model, and a second-order model. It is shown how the second-order model is formally equivalent to the testlet…

This article shows that the mean and covariance structure of the predetermined autoregressive latent trajectory (ALT) model are very flexible. As a result, the shape of the modeled growth curve can be quite different from what one might expect at first glance. This is illustrated with several numerical examples that show that, for example, a…

The Predictive Evaluation (PE) model is a training and evaluation approach with the element of prediction. PE allows trainers and business leaders to predict the results, value, intention, adoption, and impact of training, allowing them to make smarter, more strategic training and evaluation investments. PE is invaluable for companies that…

The focus of this presidential address is on the contemporary treatment of reliability and validity in educational assessment. Highlights on reliability are provided under the classical true-score model using tools from latent trait modeling to clarify important assumptions and procedures for reliability estimation. In addition to reliability,…

Multinomial processing tree models are widely used in many areas of psychology. A hierarchical extension of the model class is proposed, using a multivariate normal distribution of person-level parameters with the mean and covariance matrix to be estimated from the data. The hierarchical model allows one to take variability between persons into…

Motivated by Gibbons et al.'s (Appl. Psychol. Meas. 31:4-19, "2007") full-information maximum marginal likelihood item bifactor analysis for polytomous data, and Rijmen, Vansteelandt, and De Boeck's (Psychometrika 73:167-182, "2008") work on constructing computationally efficient estimation algorithms for latent variable…

Action research (AR) is defined by Macintyre to be: "an investigation, where, as a result of rigorous self-appraisal of current practice, the researcher focuses on a problem,or a topic or an issue which needs to be explained, and on the basis of information, plans, implements, then evaluates an action then draws conclusions on the basis of…

Longitudinal studies are permeating clinical trials in psychiatry. Therefore, it is of utmost importance to study the psychometric properties of rating scales, frequently used in these trials, within a longitudinal framework. However, intrasubject serial correlation and memory effects are problematic issues often encountered in longitudinal data.…

In studies of change in education and numerous other fields, interest often centers on how differences in the status of individuals at the start of a period of substantive interest relate to differences in subsequent change. In this article, the authors present a fully Bayesian approach to estimating three-level Hierarchical Models in which latent…

The purpose of this article is to clarify conceptions, definitions, and applications of learning and problem-solving efficiency. Conceptions of efficiency vary within the field of educational psychology, and there is little consensus as to how to define, measure, and interpret the efficiency construct. We compare three diverse models that differ…

Previous work on statistical power has discussed mainly single-level designs or 2-level balanced designs with random effects. Although balanced experiments are common, in practice balance cannot always be achieved. Work on class size is one example of unbalanced designs. This study provides methods for power analysis in 2-level unbalanced designs…

The data for 104 software projects is used to develop a linear regression model that uses function points (a measure of software project size) to predict development effort. The data set is particularly interesting in that it violates several of the assumptions required of a linear model; but when the data are transformed, the data set satisfies…

In the predictive learning and causal reasoning literature it has been suggested that the processing of events is under the control of a competitive mechanism. However, little is known about whether the competitive mechanism operates at the encoding or near the response stages. The present work suggests that measures based on the recall of…

A new system of order analysis, developed by Takeya and called Item Relations Structure Analysis (IRSA), was described and used for examining the structural relations among a set of 24 items on the addition and subtraction of fractions. A diagraph showing 16 chains of items that had discernibly common features was generated by this method, and…