In Japanese secondary schools, speaking assessment in English classrooms is designed, conducted, and scored by teachers. Although the assessment is intended to be used for summative and formative purposes, it is not regularly or adequately practiced. This paper reports the problems (i.e., lack of continuous speaking assessment, limited speaking test formats used, and reliability that is not ensured) and presents future directions for second language speaking assessment implementation to enhance the summative and formative uses of speaking assessments: improving teacher training and resources and discussing an effective speaking assessment framework.