Drawing parallels to classical test theory, this article clarifies the difference between rater accuracy and reliability and demonstrates how category marginal frequencies affect rater agreement and Cohen's kappa. Category assignment paradigms are developed: comparing raters to a standard (index) versus comparing two raters to one another…

In the current time of globalization, collaboration among people in virtual environments is becoming an important precondition of success. This trend is reflected also in the educational domain where students collaborate in various short-term groups created repetitively but changing in each round (e.g. in MOOCs). Students in these kind of dynamic…

The "TextEvaluator"® text analysis tool is a fully automated text complexity evaluation tool designed to help teachers, curriculum specialists, textbook publishers, and test developers select texts that are consistent with the text complexity guidelines specified in the Common Core State Standards.This paper documents the procedure used…

Estimating the difficulty level of math word problems is an important task for many educational applications. Identification of relevant and irrelevant sentences in math word problems is an important step for calculating the difficulty levels of such problems. This paper addresses a novel application of text categorization to identify two types of…

Formulas for estimating cross-validated hit-rates, the number of correct classifications into an a priori grouping structure, were examined. The following mathematical formulas were compared: McLachlan's formula estimator, two Snappin and Knoke smoothed formula estimators, and the analytic leave-one-out estimator. The R method was included as a…

For six tests, coefficient delta as an index for internal optimality is computed. Internal optimality is defined as the magnitude of risk of the decision procedure with respect to the true score. Results are compared with an alternative index (coefficient kappa) for assessing the consistency of decisions. (Author/JKS)