School-within-a-school programs are an alternative school choice that can provide differentiated learning opportunities for academically gifted students, but they are often politically contentious. In a recent study, we interviewed 530 students and teachers in gifted and regular streams in three publicly funded secondary schools with different approaches to high-ability school-within-a-school programs: gifted, International Baccalaureate, and science-focused. Although teachers and students across conditions expressed strong satisfaction with the academic challenges provided by the special programs, they also expressed serious concerns about the relationship between these programs and the larger schools within which they are housed. Taking into consideration concerns about gifted education usurping resources from more urgent educational and societal goals, we discuss policy implications of our findings, considering ways to "allow idiosyncratic learners to thrive" (in the words of a teacher interviewed in this study), while minimizing misconceptions, prejudices, and perceptions of elitism. (Contains 1 table.)