In November 2022, OpenAI released ChatGPT, a groundbreaking generative AI chatbot backed by large language models (LLMs). Since then, these models have seen various applications in education, from Socratic tutoring and writing assistance to teacher training and essay scoring. Despite their widespread use among high school and college students in the United States, there is limited research on students' understanding and perception of these technologies. This study aims to fill that gap by developing a novel framework for generative artificial intelligence (GenAI) literacy, focusing on what undergraduate students know about generative AI and how they perceive the capabilities of AI chatbots. We designed a GenAI literacy survey to measure students' knowledge and perceptions, collecting data from 568 undergraduate students. The results show that about 60% of students use AI chatbots regularly for academic tasks, but they often overestimate the capabilities of these tools. However, increased knowledge about how generative AI works correlates with more accurate estimation of its capabilities in real-world tasks. Our findings highlight the need for enhanced GenAI and AI literacy to ensure students use these tools effectively and responsibly. This research underscores the importance of developing educational strategies and policies that prepare students for critical and informed engagement with AI technologies.