Mathematically gifted and talented students have unique cognitive and emotional needs. Thus, their schooling should consider their social and emotional development. This study investigated student affect in a science high school in Korea to determine how the specialized school's curriculum and instruction influence student affect. Data were collected through interviews and a survey. Our findings show that gifted and talented students generally demonstrate a positive affect toward mathematics, but high anxiety and low self-concept play a key role in shaping students' negative attitudes toward mathematics. In this paper, we argue that two different goals -- gifted education and college preparation -- could cause such negative student affect. The dual educational goal manifests itself in fast-paced instruction and memorized problem solving, which can ultimately cause students to lose confidence and interest in mathematics.