In online courses, discussion forums play a key role in enhancing student interaction with peers and instructors. Due to large enrolment sizes, instructors often struggle to respond to students in a timely manner. To address this problem, both traditional machine learning (ML) (e.g., Random Forest) and deep learning (DL) approaches have been applied to classify educational forum posts (e.g., those that required urgent responses versus those that did not). However, there lacks an in-depth comparison between these two kinds of approaches. To better guide people to select an appropriate model, we aimed at providing a comparative study on the effectiveness of six frequently-used traditional ML and DL models across a total of seven different classification tasks centering around two datasets of educational forum posts. Through extensive evaluation, we showed that (1) the up-to-date DL approaches did not necessarily outperform traditional ML approaches; (2) the performance gap between the two kinds of approaches can be up to 3.68% (measured in F1 score); (3) the traditional ML approaches should be equipped with carefully-designed features, especially those of common importance across different classification tasks. Based on the derived findings, we further provided insights to help instructors and educators construct effective classifiers for characterizing educational forum discussions, which, ultimately, would enable them to provide students with timely and personalized learning support.