Large-scale numeracy assessments are intended to facilitate the improvement of educational outcomes; however, it is not clear exactly how this is to be achieved. To move towards the goal of numeracy for all, it is necessary to systematically address issues that are known to be difficult, pervasive and persistent. This paper includes an analysis of an "addition of fractions" item from the Australian 2008 Year 7 NAPLAN assessment and draws insights that may be generalised to improve overall numeracy.