Artificial intelligence (AI) is currently leading an industrial revolution in most aspects of human life, and education is no exception. With the increasing ratio of students to faculty, AI could be an extremely beneficial tool for individual mentoring; for example, for cases of dropout and for student retention. While many models have already been built, the adoption of AI in education has been lower than expected, and few interventions have emerged from those models. Several factors may be in play, but one is that AI models are not easily explained, and the lack of explanation is fatal for situations like dropout prevention. An ideal AI-based tool for this problem would provide individually tailored interventions, but that would require a much deeper understanding of what a successful intervention entails. Using a novel methodology for feature comparison between student subpopulations, we compared regular students against students under academic guidance on a dataset containing 124,000 unique students and 36 informative features. We found that the explanations obtained regarding student dropout matched the real-world experiences of mentors and tutors, especially when dealing with highly explanatory features like previous average grades and interventions.