During the COVID-19 pandemic, distance learning became the predominant teaching method at most universities, exposing students and teachers alike to novel and unexpected challenges and learning opportunities. Our study is situated in the context of higher physics education at a large Swedish university and adopts a mixed-methods approach to explore how students perceive shifts to distance learning. Quantitative student survey responses comparing distance learning during the pandemic with previous in-person learning are analyzed with k-means cluster analysis and with a random-intercept multilevel linear model. Combined analyses produce a consistent picture of students who report having experienced the greatest challenges. They are on average younger, report being less autonomous in their learning, and find it harder than peers to ask questions to the instructor. They are also less likely to have access to a place where they can study without interruptions. Variation across courses is small with students being largely subjected to the same set of challenges. Qualitative data from semi-structured focus group interviews and open-ended questions supports these findings, provides a deeper understanding of the struggles, and reveals possibilities for future interventions. Students report an overall collapse of structure in their learning that takes place along multiple dimensions. Our findings highlight a fundamental role played by informal peer-to-peer and student-instructor interactions, and by the exchange of what we refer to as "structural information." We discuss implications for teachers and institutions regarding the possibility of providing support structures, such as study spaces, as well as fostering student autonomy.