While research suggests that mathematical modeling is accessible to elementary grade students, questions remain about how to assess modeling competency. Our research addresses this need through the systemic design and empirical testing of a Mathematical Modeling Student Assessment (MMSA) for elementary grades 3 through 5. Research questions focus on validity evidence for the MMSA based on test content, response process, internal structure, and relationship to other variables, as well as reliability evidence. Data sources included student interviews, MMSA post assessments and state level standardized assessments. We paid particular attention to evidence of internal structure to understand the extent to which a holistic versus an atomistic conceptualization of mathematical modeling competency was empirically supported by the data. We found that MMSA items provided sufficient coverage to provide accurate information about a range of modeling competencies. The MMSA had high reliability and low standard error for average modeling competency which supports its use for grades 3 to 5. Correlations between MMSA trait-level estimates and standardized assessment scores indicate the MMSA captures overlapping but distinct competencies specific to modeling. We discuss implications for teaching modeling and directions for future research on the development of modeling assessments in the elementary grades.