Feeling like an impostor is common among successful individuals, but particularly among women and early-career professionals. Here, we investigated how gender and career-stage differences in impostor feelings vary as a function of the contexts that academics have to navigate. In particular, we focused on a powerful but underexplored contextual feature: the extent to which raw intellectual talent (i.e., "brilliance") is prized in an academic field. We hypothesized that gender and career-stage differences in impostor feelings would be magnified in fields that value brilliance. We tested this hypothesis using the largest sample of academics that has been brought to bear on the impostor phenomenon to date, with over 4,000 academics recruited from nine research-intensive U.S. universities and representing more than 80 fields across the natural and social sciences, the humanities, and medicine. Consistent with our hypothesis, the more that success in a field was perceived to require brilliance, the more that women--especially women from racial/ethnic groups that are traditionally underrepresented in academia--and early-career academics felt like impostors. Impostor feelings were also related to a lower sense of belonging in a field and lower self-efficacy, highlighting the potential negative implications of the impostor phenomenon for academics' long-term success and for the diversity of fields that value brilliance.