The purpose of this study was to generate an item bank for assessing faculty members' assessment literacy and to examine the applicability and feasibility of a Computerized Adaptive Test (CAT) approach to monitor assessment literacy among faculty members. In developing this assessment using a sequential mixed-methods research design, our goal was to create a simple, quick, and precise screening tool for assessing literacy in higher education. After defining the construct of assessment literacy within the higher education context, we developed the test blueprint and items, and subjected them to a series of expert reviews. Following a pilot administration to confirm feasibility, we conducted item parameter calibration using a representative sample of faculty members (n = 211) selected through a convenience sampling approach. We evaluated the items for evidence of adequate psychometric quality, including fit, targeting, and unidimensionality under the Rasch framework. We concluded that developing an adaptive test for measuring assessment literacy is possible even with a small item pool and a small calibration sample.