Although portfolio assessment is becoming increasingly popular, it may not survive unless portfolio scoring can meet the demands of large-scale assessment standards. The results of studies of interrater reliability with large-scale portfolio assessments have been mixed. This paper reports the scoring results of a nationwide portfolio pilot in which over 2,000 secondary students submitted portfolios from language arts, mathematics, and science classes. For language arts, both interrater reliability and score reliability were at reasonable levels. For mathematics, the interrater reliability was adequate, but the score reliability was low. For the science portfolio, neither the interrater reliability nor the score reliability was adequate. Generalizability studies also suggest that adequate reliability for student level decisions can be achieved with scores derived from five portfolio entries, each scored by two raters. With changes to the scoring rubrics and student and teacher manuals, more reliable scores should result in the second year of the project. (Contains nine tables and nine references.) (Author/SLD)