This paper examines the insistent claims by advocates of evidence-based teaching that it is a rigorous scientific approach. The paper questions the view that randomised controlled trials and meta-analyses are the only truly scientific methods in educational research. It suggests these claims are often based on a rhetorical appeal which relies on too simple a notion of "science". Exploring the tacit assumptions behind "evidence-based teaching", the paper identifies an empiricist and reductionist philosophy of science, and a failure to recognise the complexity of education and pedagogy. Following a discussion of large-scale syntheses of evidence (Hattie's Visible Learning; the Education Endowment Foundation's Teaching and Learning Toolkit), it examines in detail one strand of the latter concerning sports participation, which is used to illustrate flaws in procedures and the failure to take seriously the need for causal explanations.