Background: The Progress in International Reading Literacy Study (PIRLS), and similar international assessments, have consistently shown South African intermediate phase learners' performance to be among the lowest worldwide. Of particular concern is the Curriculum and Assessment Policy Statement (CAPS) for Home Language in the Intermediate Phase and, specifically, the document's treatment of the assessment of reading comprehension. Objectives: In this study, the CAPS requirements for assessing reading comprehension were examined, with the aim of laying the groundwork for an improved policy framework. Method: The research design of the study involved evaluating the assessment of reading comprehension in the CAPS document, using a social realist approach to uncover its underlying structures and mechanisms. Results: The study found that a principled approach to the assessment of reading comprehension was lacking, which had a cumulative effect across the CAPS document, resulting in random, yet highly prescriptive, requirements. Conclusion: The study concluded that CAPS does not provide adequate guidance for improving reading comprehension and, moreover, that the prescribed programme of assessment is not supported by the research literature on reading comprehension. The study recommends that better, more evidence-informed and consultative policies and guidelines be introduced to support teachers in the assessment -- and, ultimately, the improvement -- of intermediate phase learners' reading comprehension.