More aggregate data on school performance is available than ever before, opening up new possibilities for applied researchers interested in assessing the effectiveness of school-level interventions quickly and at a relatively low cost by implementing comparative interrupted times series (CITS) designs. We examine the extent to which effect estimates conducted using a CITS approach correspond to a randomized trial, varying (1) the approach to modeling the pre-intervention trend, (2) the approach used to select comparison cases, and (3) the number of years between the implementation of the intervention and measurement of the outcome of interest. Our analyses demonstrate that CITS can come close to replicating experimental findings. However, analytic decisions matter. Picking a comparison group that accounts for local context, past performance, and school demographics can reduce bias and reliance on model selection. Likewise, fitting nonlinear trend models can lead to substantial bias in short time series analyses. Finally, the performance of CITS declines for outcomes measured further after the introduction of the intervention.