Texts used in L2 classrooms have traditionally been simplified manually, but recent technological advances allow us to investigate whether automatic text simplification (ATS) software can help L2 learners comprehend texts in second and foreign languages. Participants were divided into low and high L2 reading proficiency groups and assigned to read either the authentic or automatically simplified version of a text and completed a free recall task and MC comprehension test. The results did not show any significant correlations among the variables of topic knowledge, topic interest, and MC comprehension, but there were correlations among L2 reading comprehension, MC comprehension, and free recall results. Results also showed that the automatically simplified text facilitated the comprehension of the more proficient readers but not the less proficient readers according to their performance on the free recall assessment. Implications are that L2 teachers cannot blindly use whatever text they want with ATS, and ATS software designers may need to reconsider the current conservative approach to simplification that many ATS tools use.