This article outlines the development of a 16-item instrument for measuring language learner's foreign language self-efficacy (SE) concerning their speaking and listening skills through repeated administrations to groups of Japanese tertiary students. Responses were analysed through the Rasch model, which allows researchers to investigate unidimensionality of each proposed component, as well as the functioning of individual items. Logit scores from Rasch were used for a regression analysis of listening SE with listening proficiency test data, and a correlation study with speaking SE and speaking proficiency scores of a subset of participants. The results suggest that the two proposed components in the instrument are unidimensional and are able to differentiate between different levels of SE among participants. Also, there are low to moderate correlations between learners' aural/oral SE and their language proficiency.