Background: People with intellectual disabilities (IDs) and their caregivers face difficulties during the COVID-19 pandemic. However, limited studies have comprehensively investigated their challenges, especially in Japan. We aimed to clarify the concerns and needs of people with IDs and their caregivers during the COVID-19 pandemic in Japan. Method: From March to August 2021, 27 in-depth interviews were conducted with principal caregivers of people with moderate to profound IDs in Japan. We then transcribed the interviews and conducted deductive coding using predetermined codes focused on their daily life difficulties. Inductive coding was used to ensure that no important themes were overlooked. Results: We found four concerns and four needs among people with IDs and their caregivers as significant themes. Conclusions: Our results provide useful information for supporting people with IDs and their caregivers, especially among those who need medical or social care in accordance with the infection control and social-distancing policies.