The National College English Test Band 4 (CET-4) is a key test to assess the English language ability of Chinese university students, and the success rate of the test is important to improve the quality of their English learning. Artificial intelligence technology can be used to predict and explore the factors influencing the success rate. This study employed machine learning techniques to analyse a dataset collected from undergraduate students at a full-time university in China who were not majoring in English. The aim of this study is to identify the most appropriate machine learning model for predicting CET-4 success and to understand the factors that most influence this success. These findings are expected to help educators improve their teaching strategies. The research found that LightGBM achieved the highest accuracy rate of 97.04% in predicting whether students could pass CET-4. Further interpretability analysis of LightGBM identified three primary factors that play a significant role in determining students' success in the CET-4 exam: their interest in English learning, GPA performance, and the experience of preparing for or participating in other types of English exams. These findings are closely related to students' learning motivations, choices, and optimization of learning strategies, as well as knowledge transfer and other psychological aspects of learning. Additionally, they are closely tied to the current educational environment in China.