A pilot essay was included in the 1985 Spring and Fall administrations of the Medical College Admission Test. A sample of 320 of the essays written by Fall examinees who had expressed an interest in allopathic medicine was used to calculate interrater reliability estimates. Sixteen of 20 readers who had been trained by White's suggestions for holistic scoring were included in the reliability study. Each reader rated two batches of 20 essays on the first day and again on the second. Analyses of variance were computed for (1) day; (2) batch; (3) reader; (4) subjects nested within batch; (5) interaction of day with batch; (6) interaction of day with reader; (7) interaction of reader with batch; and (8) interaction of subjects within batch with day. The interrater reliability index was estimated at .84, indicating that 71% of the variation in scores was due to level of differences between the essays themselves. The remainder was due to batch, day, day by reader, day by batch, batch by reader, and reader. When extrapolated to the operational setting, the index was .72. Methods of limiting the effects of these factors are proposed. (MGD)