A total of 169 Chinese college students were divided into high proficiency and low proficiency learners according to a College English Test, and watched one of three short fully captioned English videos, thus producing six groups: (1) High proficiency + L2 (n = 25), (2) High proficiency + L1 + L2 (n = 24), (3) High proficiency + L1 (n = 33), (4) Low proficiency + L2 (n = 29), (5) Low proficiency + L1 + L2 (n = 30), (6) Low proficiency + L1 (n = 28). The participants' retention and transfer were assessed first. SPSS analyses revealed that L1 + L2 caption group with high proficiency performed the best retention, while L2 caption group with low proficiency achieved the highest retention, and there was no significant difference in transfer. Afterwards, a factor design was employed and three dimensions of student engagement were evaluated with the same participants, suggesting significant results in cognitive student engagement with a median effect size, represented by investment in learning and positive attitude as indicators. This research attempts to explore the incorporation of behavioral, emotional, and cognitive student engagement when measuring learning effectiveness of content-based instruction video learning, highlighting the significance of cognitive student engagement in the studies of captioned videos.