This study explores the culture of physics departments in Sweden in relation to physics teacher education. The commitment of physics departments to teacher education is crucial for the quality of physics teacher education and the way in which physics lecturers talk about teacher education is significant, since it can affect trainees' physics learning and the choice to become a physics teacher. We analyzed interviews with eleven physicists at four Swedish universities, looking for assumptions in relation to teacher training that are expressed in their talk. We found five tacit assumptions about physics teacher training, that together paint a picture of trainee physics teachers moving in the "wrong" direction, against the tide of physics. These are the Physics Expert Assumption: the purpose of all undergraduate physics teaching is to create physics experts. The Content Assumption: the appropriate physics content for future school physics teachers is the same as that for future physicists. The Goal Assumption: the role of a school physics teacher is to create new physicists. The Student Assumption: students who become physics teachers do not have the ability to make it as successful physicists. The Teaching Assumption: If you know physics then it's not difficult to teach it. We suggest that these five assumptions, if perpetuated without reflection, risk working against high quality physics teacher education. For physics teacher educators, our results can be used as a lens to reflect on the local departmental culture and its effect on teacher education.