COVID-19 has been one of the most significant disruptors of higher education in modern history. Higher education institutions rapidly transitioned to Emergency Remote Teaching (ERT) in mid-to-late March of 2020. The extent of COVID-19's impact on teaching and learning, and the resulting challenges facilitating ERT during this time, likely varied by faculty, institutional, and geographical characteristics. In this study, we identified challenges in teaching and learning during the initial transition to ERT at Predominantly Undergraduate Institutions (PUIs) in the Midwest, United States. We conducted in-depth interviews with 14 faculty teaching at Midwestern PUIs to explore their lived experiences. We describe the most overarching challenges related to faculty teaching through four emergent themes: pedagogical changes, work-life balance, face-to-face interactions, and physical and mental health. Five themes emerged that we used to describe the most overarching challenges related to students and their learning: learning patterns, technology access, additional responsibilities, learning community, and mental health. Based upon the identified challenges, we provide broad recommendations that can be used to foster a more successful transition to ERT in unforeseen regional or global crises in the future.