Mainstream physics teaching and learning produces material outcomes that, when analyzed through the lens of Critical Race Theory, point to white supremacy, or "the systemic maintenance of the dominant position that produces white privilege" (Battey & Levya, 2016). In particular, the continued, extreme underrepresentation of People of Color in physics and a growing number of first-person accounts of the harm that People of Color experience in physics classrooms and departments speak to a system that valorizes whiteness and marginalizes People of Color. If we take Critical Race Theory as a lens, we expect that maintaining white supremacy in physics happens in part via discipline-specific instantiations of broader mechanisms that reproduce whiteness. In this study, we illustrate one such mechanism: race evasiveness, a powerful ideology that uses race-neutral discourse to explain away racialized phenomena, evading race as a shaping force in social phenomena. We offer examples from interviews with twelve university physics faculty, showing what race-evasive discourses can look like in physics and how physics epistemologies, discourses, and stories reify race-evasive frames. This work aims to support faculty in "refusing" race evasiveness in physics teaching and learning, toward developing "race-conscious" analyses that can help us challenge white supremacy in our discipline.