One of the effects of the COVID-19 pandemic was the rapid shift to replacing traditional, paper-based tests with their computer-based counterparts. In many cases, these new modes of delivering tests will remain in place for the foreseeable future. In South Africa, the National Benchmark Quantitative Literacy (QL) test was impelled to make this rapid shift. While test-makers attempted to ensure comparability between the computer-based and paper-based versions, some scholars have questioned the efficacy of these efforts. The International Test Commission (2005) declared that the equivalency of test scores obtained from computer-based and conventional paper-based testing should be established to ensure that the computer-based version is valid and reliable. This paper undertakes a quantitative comparison between these two modes of delivery using the measures of frequency, central tendency, dispersion, and position. It further uses the median test (a non-parametric test) to explore whether there are statistical differences in the median scores between the two modes of delivery. Despite the difference in the mode of delivery, the median test indicated that the cohorts in the computer-based and paper-based QL tests are drawn from a population with the same median.