One of the many challenges facing Japanese university students studying English is the multi-word phrase. The English language contains a large number of such multiple-word items, which act as single words with a single meaning. This study is concerned with evaluating the efficacy of yes/no checklist tests to assess knowledge of multi-word units. Participants (n = 206) took a yes-no test of 30 real words and 15 pseudowords. The 30 real words were selected from the students' textbook, based on the teacher's intuition of the words and multi-words posing the greatest learning burden for the students. Twenty-one of the selected words were single-word items. The remaining nine were multi-words, such as "get up" and "take turns". Forty-five minutes following completion of the yes-no test, an English to Japanese translation test of the same 30 real words was taken by the same participants to evaluate the efficacy of yes/no test. Results suggest that the yes-no vocabulary test format may be able to measure student knowledge of multi-word lexical units as (or more) effectively than single-word units.