The quality assurance and evaluation of primary schools requires early risk detection. This is a daunting task, not only because risks are typically rare and their origins complex, but also because governing institutions have limited resources and capacity and desire efficiency and proportionality. Many countries, including most Organisation for Economic Co-operation and Development countries, use inspections to detect schools at risk. In order to aid these efforts, we first evaluate various case-based prediction models, and then propose a principled exploit-explore procedure for organizing school inspections. We demonstrate these methods using data from the Dutch Inspectorate of Education, which monitors the regulatory compliance of roughly 6,000 primary schools. The approach has the potential to balance the benefits of prioritizing inspections of predicted high-risk schools on the one hand, with the benefits of verifying predicted risks and causal impact evaluations of school inspections on the other hand.