Massive open online courses (MOOCs) provide numerous open-access learning resources and allow for self-directed learning. The application of big data and artificial intelligence (AI) in MOOCs help comprehend raw educational data and enrich the learning process for students and instructors. Thus, we created two deep neural network models. The first model predicts learning outcomes on the basis of learning behaviors observed when students watch videos. The second is a novel exercise-based model that predicts if a student will correctly answer examination questions on relevant concepts. The study data were collected from two courses conducted on the National Tsing Hua University's MOOCs platform. The first model accurately evaluated student performance on the basis of their learning behaviors, and the second model efficiently predicted student performance according to how they answered the exercise questions. In conclusion, our AI system remedies the present-day inability of MOOCs to evaluate student performance. Instructors can use the systems to identify poor-performing students and offer them more assistance on a timely basis.