Many studies that have investigated the educational value of online machine translation (MT) in second language (L2) writing generally report significant improvements after MT use, but no study as of yet has comprehensively analyzed the effectiveness of MT use in terms of various measures in syntactic complexity, accuracy, lexical complexity, and fluency (CALF). The present study examined how learners' use of MT affects CALF measures in L2 writing using evaluations by automated computational tools as well as human raters. In addition, the study investigated whether proficiency level and text genre affect the learners' use of MT. A total of 91 Korean learners of English participated in the main task of the study in which they wrote on an assigned topic in English first without the help of any resource, and then on a different topic using only "Google Translate" a week later. Text analysis of students' writing revealed major improvements in accuracy but unclear benefits in syntactic and lexical complexity. It was also found that MT use provided different advantages and disadvantages depending on the proficiency level (high vs. low) and text genre (narrative vs. argumentative). Survey responses strongly indicated that students are highly satisfied with MT and plan to use it again in the future despite being aware of its limitations. Overall, this study found that MT can be useful for improving accuracy but must be used with much discretion for it to benefit other aspects of L2 writing.