Recent research focusing on the sources of listening anxiety identified individual and external factors (IEF), listeners' control over input (LC), and perceived level of task difficulty (PTD) as the main causes of anxiety among L2 listeners. This correlational study aims to explore whether the sources of anxiety in L2 listening have a causal relationship to listeners' self-efficacy and listening proficiency. Data were gathered from 347 students studying in the preparatory program of English in Turkey through Foreign Language Anxiety Scale (FLAC), Listening Self-efficacy Scale (LSS), and listening test scores of the students. Structural equation modeling (SEM) technique was conducted for data analysis. Results showed that there were strong negative associations between individual and external factors causing anxiety and self-efficacy while there were moderate level, negative associations between the perceived level of task difficulty and self-efficacy in L2 listening. Our work further proved that the listeners' control over input was positively and moderately related to the levels of listeners' self-efficacy, which was positively and strongly correlated with proficiency in L2 listening.