Modelling roles of mathematics in physics has proved to be a difficult task, with previous models of the interplay between the two disciplines mainly focusing on mathematical modelling and problem solving. However, to convey a realistic view of physics as a field of science to our students, we need to do more than train them to become fluent in modelling and problem solving. In this article, we present a new characterisation of the roles mathematics plays in physics and physics education, taking as a premise that mathematics serves as a constitutive structure in physics analogous to language. In doing so, we aim to highlight how mathematics affects the way we conceptualise physical phenomena. To contextualise our characterisation, we examine some of the existing models and discuss aspects of the interplay between physics and mathematics that are missing in them. We then show how these aspects are incorporated in our characterisation in which mathematics serves as a foundation upon which physical theories are built, and on which we may build mathematical representations of physical information that in turn serve as a basis for further reasoning and modifications. Through reasoning processes mathematics also aids in generating new information and explanations. We have elucidated each of these roles with an example from the historical development of quantum physics. To conclude, we discuss how our new characterisation may aid the development of physics education and physics education research.