One of the many ways issues of underrepresentation appear in the physics classroom is female students frequently have a lower perception of their performance and ability than their male peers. Understanding how classroom experiences impact students' confidence, especially for underrepresented students, can provide an important guide to designing physics classrooms where every student sees themselves as capable of learning and doing physics. To explore these issues in my AP Physics 1 classroom, I started asking my students to self-assess as part of my assessment process, allowing me to collect data comparing students' perceptions to their actual performance. I also conducted interviews and collected student reflections to gain insights into the classroom experiences that impacted students' confidence in physics. My students made it clear that discovering concepts in the lab contributed to their confidence. Girls also built confidence from teacher feedback, even on assessments where they scored poorly, while boys saw peer interactions as a source of confidence.