The recent interests in research in the assessment field have been rapidly shifting from decision-maker-centered assessments to learner-centered assessments (i.e., diagnostic and/or formative assessments). In particular, it is a very important research topic in this field to analyze how these learner-centered assessments are developed more practical and valid by combining information (or intelligent) and communication technologies (ICT) and psychometric advances, and how these systems contribute to learning. Automatic Item Generation (AIG), which is in the spotlight recently, is a representative example of building a next-generation assessment theory and practice by integrating ICT into psychometrics. AIG has very broad and promising features in mass production, intelligent item calibration and management, especially learner-centered assessment, and so on. However, these claims have not yet been fully validated in real education settings, and it is in dire need of evaluating the utilities of AIG applications in practical educational settings. Based on these needs, the purposes of this research are, firstly, to introduce the benefits and challenges of AIG to properly assess AIG's utilities. And, secondly, this study provides empirical evaluation results of AIG's utilities within a formative assessment system for Korean high school students and teachers. The results of this study have important implications in that it is the first empirical study to evaluate the usefulness of AIG in the formation evaluation application of the actual educational environment, Korean high school setting. The results of the theoretical and empirical evaluation of the AIG will also be useful to researchers and practitioners who wish to evaluate the benefits and issues of future AIG-based educational services or theory development.