Transitions between educational levels have been identified as posing potential barriers for students' sense of belonging in science. In this paper, we focus on the transition from lower secondary to upper secondary school while foregrounding physics as a subject. We approach transitions as an ongoing negotiation-process of identities embedded within the norms, practices, and expectations of physics. Methodologically, we narrow the focus to students who self-identify as female, as these are underrepresented in physics worldwide. We apply the analytical lens of physics identity constructed through competence, performance, interest, and recognition while drawing on the concept of the ideal student to understand what identities are idealized and marginalized, and how these are negotiated by the students. We found that a large group of the students were marginalized in relation to their interests in physics and experienced learning physics to be instrumental and meaningless. Only a small group were able to form a sense of identity mainly due to the resemblances of physics to mathematics. In conclusion we call for attention on who counts as knowledgeable and what counts as knowledge in the physics classroom. Second, we wish to question interests as imperative for being in physics. Lastly, we urge reflection on mathematics' role in physics and what physics is without mathematics to disrupt the elite status of the subject.