In this study, we investigated the impact of the evaluation capacity building (ECB) efforts of an organization by examining the evaluation plans included in funding proposals over a 14-year period. Specifically, we sought to determine the degree to which and how evaluation plans in proposals to one National Science Foundation (NSF) program changed over time and the extent to which the organization dedicated to ECB in that program may have influenced those changes. Independent raters used rubrics to assess the presence of six essential evaluation plan elements. Statistically significant correlations indicate that proposal evaluation plans improved over time, with noticeable differences before and after ECB efforts were integrated into the program. The study adds to the limited literature on using artifacts of evaluation practice rather than self-reports to assess ECB impact.