This study introduces the first Arabic Vocabulary Levels Test (Arabic-VLT), created for foreign learners of Arabic. We present compelling evidence to substantiate its validity and reliability. The Arabic-VLT was developed according to five levels, beginning with the most frequently used words (Level 1) to the least frequently used ones (Level 5), taken from a reliable and well-known Arabic wordlist. Each level has 30 correct answers with a total of 150 marks. The Arabic-VLT was developed and assessed based on item response theory (IRT), specifically through the Rasch model following aspects of Messick's validation framework. We found that the items of the Arabic-VLT showed good reliability and validity via the Rasch model. In addition, students' scores on the test correlated positively with the period they spent learning Arabic, which supports the criterion validity of the Arabic-VLT. The introduction of Arabic-VLT as the first valid vocabulary knowledge test opens new avenues for experimental research in Arabic as an L2 language. We believe that researchers and practitioners can benefit from and replicate our findings.