A study investigated the comparability of second language (L2) listening comprehension tests by examining the effects of varying text type and assessment task on student performance. Student target language experience was an additional variable considered. Subjects were 107 beginning and 64 advanced-intermediate college-level Spanish second language learners. Stimulus materials were a lecture text and an interview text on the same topic and with comparable content. The assessment tasks were a multiple-choice task, an open-ended task, and a cloze task, each consisting of 10 items. Tasks were randomly assigned. Relationships between text type, assessment task type, score, and language experience were analyzed. Results indicate that, contrary to previous research findings, text type alone was not a significant factor in L2 listening comprehension performance. However, assessment task was a significant factor in performance, with multiple-choice tasks producing the highest scores. Students with more target language experience had higher scores on all evaluations, but language experience did not change the relative performance on different task types. With one exception, the three variables did not interact. The single interaction occurred between test type and assessment task on items testing comprehension details. Several recommendations for instruction and testing resulting from this study are offered. (MSE)