"Dalits" are a historically marginalized caste group in India. In this study, we examine the impact that the institution of caste has on career change. Focusing on the shift from science, technology, engineering, and mathematics (STEM) to non-STEM careers, we ask why members of the Dalit communities in India leave their STEM careers. To present a comprehensive understanding of this career change, we also examine the factors that facilitate their retention in STEM careers. Deriving data from in-depth interviews with 42 STEM scholars from the Dalit communities in India, we illustrate their organizational experiences that are based on casteism. Qualitative data analyses were done using the grounded theory method. Results suggest that while caste operates in a unique way to drive away Dalit scholars from their STEM careers, doing gender plays an important role in their retention within the same careers.