This study investigated the performance of four widely used data-collection designs in detecting test-mode effects (i.e., computer-based versus paper-based testing). The experimental conditions included four data-collection designs, two test-administration modes, and the availability of an anchor assessment. The test-level and item-level results were analyzed with inferential statistics and multidimensional scaling, respectively. The test-level results supported the superiority of the single-group counterbalanced design and the random-groups design over the single-group design without counterbalancing and the anchor test design in the recovery of the actual test-mode effects. Analysis at the item level revealed the presence of a two-dimensional solution, with the data-collection design contributing significant variability over and beyond the test mode.