Four discrimination indices were compared, using score distributions which were normal, bimodal, and negatively skewed. The score distributions were systematically varied to represent the common circumstances of a military training situation using criterion-referenced mastery tests. Three 20-item tests were administered to 110 simulated subjects. The cutting score on each test was 10 items correct. Three databases were constructed for normal, bimodal, and skewed score distributions. Five item analysis statistics were calculated: the p statistic, two versions of the upper-lower group statistics, the phi coefficient, and the point-biserial correlation. Analysis of variance and t-tests were used to estimate differences between the discrimination index values. With normal data, the second upper-lower statistic produced the largest discrimination values, point-biserial next, and phi coefficient and the first upper-lower produced identical, least discriminating values. Similar results were obtained for the bimodal discrimination indices. The skewed distribution analysis was slightly different, with the first upper-lower results larger than the phi coefficients. The second upper-lower method was not significantly different from the point-biserial correlation. (Suggestions for choosing a method are summarized in a matrix and a decision tree). (GDC)