Purpose: This study aims at assessing item fairness in students' evaluation of teaching based on students' academic college using measurement invariance analysis (MI). Design/methodology/approach: The sample of this study consists of 17,270 undergraduate students from 12 different academic colleges. SET survey consists of 20 Likert-type items distributed to four factors: planning, instruction, management and assessment was used to collect the data. The Lavaan R package with confirmatory factor analysis (CFA) was used to evaluate measurement invariance (MI). Four models of CFA were investigated and assessed: the configural model, the metric model, the scalar model and the residual invariance model. ANOVA was used to test the differences in SET according to academic colleges. Findings: MI analysis showed that the four levels of MI models are supported. ANOVA test showed that means of SET total scores are statistically different according to students' academic colleges. College of "Education" has the highest SET mean (88.64 out of 100), and all the differences between the College of Education's SET mean and other colleges' SET means are statistically significant. Practical implications: The study recommends that higher education institutions test the MI of SET according to academic colleges and then use colleges with the highest SET at the university level as internal benchmarking to develop and enhance their teaching practices. Originality/value: This study is probably the only study that tested MI according to students' colleges before testing the differences between colleges in SET. If MI is not supported, then the comparisons between academic colleges are not applicable.