It is common practice for assessment programs to organize qualifying sessions during which the raters (often known as "markers" or "judges") demonstrate their consistency before operational rating commences. Because of the high-stakes nature of many rating activities, the research community tends to continuously explore new…

Stan is a new Bayesian statistical software program that implements the powerful and efficient Hamiltonian Monte Carlo (HMC) algorithm. To date there is not a source that systematically provides Stan code for various item response theory (IRT) models. This article provides Stan code for three representative IRT models, including the…

Sixteen measures of reliability for two-category nominal scales are compared. Upon correcting for chance agreement, there are only five distinct indices: Fleiss's modification of A-sub-1, the phi coefficient, Cohen's kappa, and two intraclass coefficients. Recommendations for choosing an agreement index are made based on definitions, magnitude,…

An artifact encountered in regression models of human judgment is explored. The direction and magnitude of the artifactual effect is shown to depend upon the nature of the experimental task and task conditions. Use of an alternative index is recommended. (Author/JKS)

A rederivation of Lord's formula for estimating variance in multiple matrix sampling is presented as well as the ways Cronbach's coefficient alpha and the Spearman-Brown prophecy formula are related in this context. (Author/JKS)

Whether the one- or two-factor Wechsler or weighted sum scores (WSS) structural models adequately fit the Wechsler Intelligence Scale for Children III (WISC-III) covariance matrices was studied with samples of 2200 and 1118 children from previous studies. Both the WSS and supplemented models provided better fit than Wechsler's model but still…

Two procedures for Rasch, sample-free item calibration are reviewed and compared for accuracy. The theoretically ideal "conditional" procedure is impractical for more than fifteen items. The more practical but biased "unconditional" procedure is discussed in detail. (Author/JKS)

A method based on the Poisson approximation to the binomial distribution and on the relation between the Chi-Squared distribution and the Poisson distribution is suggested for selected use in determining the number of items and passing scores in mastery Lests. (Author/RL)

Twelve variance-analytical and nonparametrical coefficients of reliability for rating scales designed for rating persons were compared to each other theoretically and empirically. Preference for two coefficients was established. The intraclass correlation coefficient appeared to be useful for the estimation of reliability as well. (Author/RL)

A formula that is simpler to calculate than the Kappa statistic of J. Cohen is presented for the situation where each subject in an experiment is rated on a nominal scale by two or more judges. Equivalence with Pearson's chi-square statistic in this situation is demonstrated. (SLD)

The present study compares the robustness of two different one way fixed-effects analysis of covariance (ANCOVA) models to investigate whether the model which uses a test statistic incorporating estimates of separate unequal regression slopes is more robust than the conventional model which assumes the slopes are equal. (Author/BW)

The utility of the fail-safe "N" statistic was evaluated by computing it for studies in three organizational research domains in which discrepant conclusions were reached by initial and subsequent meta-analyses. Calculation of the fail-safe "N" may have led to more cautious interpretations. Implications for meta-analyses are…

Relationships among individual- and group-directed measures of leader behavior descriptions and five variables were examined using 54 law enforcement agency personnel associated with a large public university. Data on a questionnaire completed by participants during an interview were studied. Explicit consideration was given to multiple levels of…