How smartness is defined within schools contributes to low academic achievement by poor and racial/ethnic minority students. Using Holland et al.'s (1998) [Holland, D., Lachicotte, W., Skinner, D., & Cain, C. (Eds.) (1998). "Identity and agency in cultural worlds." Cambridge, MA: Harvard University Press.] concept of "figured worlds," this paper explores the "figuring" of smartness through the perspectives of marginalized youth. The youth made key distinctions between being book smart vs. street smart. This distinction is a direct challenge by the youth to the dominant discourse of smartness or "book smarts" as it operates in schools. To the youth, "street smarts" are more important because they are connected to being able to maneuver through structures in their lives such as poverty, the police, street culture, and abusive "others." This distinction is key because street smarts stress agency in countering social structures whereas, for many of the youth, book smarts "represented" those structures, such as receiving a high school diploma. Implications for schools and pedagogy are discussed.