This study examines how listening comprehension difficulties (LCDs) vary among college-level learners of English as a Foreign Language (EFL), with a focus on the roles of gender and language proficiency. Listening comprehension is an essential component of language learning, presenting notable challenges for EFL students. These challenges are influenced by specific learner characteristics, including gender and language proficiency level, which impact both the learning process and comprehension ability. The objective of the study is to understand the interaction between these factors and LCDs, especially in the college student population. The research was conducted at a university in Taiwan, involving 304 students in a survey. The results reveal a subtle but significant influence of both gender and language proficiency level on students' perceptions of LCDs. Although the study did not find gender differences to significantly alter students' perceptions of LCDs overall, it was noted that female students experienced more difficulty in processing listening inputs than their male counterparts. Moreover, students with lower language proficiency levels reported facing more challenges in listening comprehension than those with higher proficiency. Interestingly, more advanced learners indicated slightly more difficulty with contextual aspects of listening. While this study did not find statistically significant differences attributable to gender and English proficiency, it highlights the importance of acknowledging the subtle variations in student perceptions. This understanding allows educators to create suitable instructional strategies to improve listening proficiency across all students, irrespective of their gender or English proficiency level.