Kindergarten entry assessments (KEAs) are frequently used to understand students' early literacy skills. Amidst COVID-19, such assessments will be vital in understanding how the pandemic has affected early literacy, including how it has contributed to inequities in the educational system. However, the pandemic has also created challenges for comparing scores from KEAs across years and modes of administration. In this study, we examine these issues using a KEA administered to most Kindergarten students in Virginia. This screener was rapidly converted to an online platform to ensure students could continue taking it during the pandemic. Results indicate that the sample of students taking the test shifted substantially pre- and post-pandemic, complicating comparisons of performance. While we do not find evidence of noninvariance by mode at the test level, we do see signs that more subtle forms of item-level bias may be at play. Implications for equity, fairness, and inclusion are discussed.