In STEM education, grit is increasingly the focus of research, with scholars and educators seeking to develop and test interventions that will enhance persistence. As part of the special issue Centering Affect and Emotion Toward Justice and Dignity in Science Education, in this paper, we use interviews with 12 white physics faculty to show that physics culture has taken up the narrative that grit is key to success in the discipline. Using affective technology (Zembylas and Leonardo, 2013), habitus (Bourdieu, 1972/1977) and emotional habitus (Gould et al. 2019) as theoretical anchors, our analysis shows that grit, as described by faculty participants, is part-and-parcel of a "white physics habitus." In other words, grit acts to reproduce systems of dominance through the internalization of a set of structures, symbols, and worldviews that produce embodied, affective responses, drawing dominant actors toward particular embodiments of hard work and turning them away from others. Thus, we argue that power in physics is mediated through affect and embodiment. Employing qualitative case study methods, we theorize how "whiteness," in part, is reproduced in the discipline--how physics came to and continues to be a discipline where power is concentrated in the bodies of white males. We end by joining with existing calls to refuse grit, building from the work of STEM Scholars of Color who have called attention to the "suffering" that is endemic to notions of schooling and school science.