This study aims to evaluate the online measurement and evaluation course in teacher training programs during the COVID-19 process. In the study, we sought answers to two primary research questions: What are the opinions of teachers and school administrators regarding their measurement and evaluation competencies? Does the online "measurement and evaluation" course have the qualities of an effective program in the "antecedents, transactions, and outcomes" dimension? We structured the research into two phases within a multistage evaluation design framework. The findings show that there were problems and positive aspects in all dimensions of the program. For example, adapting teacher training programs developed before COVID-19 to distance education processes was challenging. In distance education, some practices contradict the modern teaching and assessment approach. Such problems were reflected in teachers' acquisition of measurement and evaluation competencies. The achievement test we applied to the observed groups also confirmed these findings. For this reason, responsible organizations should not ignore the fact that we cannot renounce distance education. During program development, they should reconsider how the teachers will acquire measurement and evaluation competencies and how we will measure and evaluate in distance education.