The National Assessment of Educational Progress (NAEP) collects data in the form of repeated, discrete measures (test items) with hierarchical structure for both measures and subjects, that is complex by any standard. This complexity has been managed through a "divide and conquer" approach of isolating and evaluating sources of variability one at a time, using a sequence of relatively simple analyses. The cost of this simplicity for the NAEP has been limits on the propagation of information from one subanalysis to another. This has made some questions that would be relatively straightforward to address in ordinary circumstances, quite difficult to answer for the NAEP. This study considers NAEP's fragmented analysis of errors in the rating of open-ended responses, develops methodology for more unified analyses, and applies the methodology to the analysis of rater effects in NAEP data. How to minimize rater effects using modern imaging technology is studied, and conclusions and recommendations are drawn in light of these analyses. (Contains 15 figures, 13 tables, and 30 references.) (SLD)