School closures in Indonesia during the COVID-19 pandemic have left 45.5 million school students and 3.1 million teachers dependent on online teaching and learning. Online teaching and learning are an unprecedented experience for most teachers and students; consequently, they have a limited experience with it. This paper examines the views of secondary school mathematics teachers on E-learning implementation barriers during the COVID-19 pandemic at four barrier levels, namely teacher, school, curriculum and student. Furthermore, it assesses the relationship between barrier levels with teachers' demographic background. Data was collected through an online questionnaire, involving 159 participants from lower and upper secondary schools in Indonesia. The findings of this study suggest that student level barrier had the highest impact on e-learning use. In addition, the student level barrier showed strong positive correlation with the school level barrier and curriculum level barrier. The study showed that teachers' backgrounds had no impact on the level of barriers. This study stimulates further discussion on the way to overcome e-learning barriers whilst simultaneously maximizing benefits of E-learning during this pandemic and beyond it by highlighting the importance of students' voices.