Although a variety of learning benefits of peer assessment have been documented, many concerns remain regarding its reliability and validity, especially for peers at early learning stages and for English as a foreign language (EFL) learners in general. Such concerns may prevent adoption of peer assessment for instruction. To help localize the appropriate contexts for peer assessment, this study examined whether reliability and validity of peer assessment changes over years in a program either for overall scores or specifically for high-level dimensions or language conventions. Participants were 118 English major undergraduates in a comprehensive university in Northeast China. We found that the peer assessments for both 1st year and 4th year students had high reliability, similar to reliability levels shown by teachers. However, the validity of peer assessment showed developmental growth: the peer assessments of 1st year EFL students had low validity, especially for assessments of language conventions, but relatively strong validity of assessments of higher-level dimensions in both 1st and 4th year students. This study suggests peer assessment may be valid for assessments of higher-level concerns across broad developmental levels, but may require stronger supports when used for assessing lower levels of language, particularly for students early on in their language development.