As we move into the 21st century, educationalists are exploring the myriad of possibilities associated with Computer Based Assessment (CBA). At first glance this mode of assessment seems to provide many exciting opportunities in the mathematics domain, yet one must question the validity of CBA and whether our school systems, students and teachers are ready to harness this form of assessment. The most obvious advantages of CBA are the speed and accuracy of accessing results and the opportunities for innovative item development. This paper will aim to highlight how several factors can obstruct the validity and reliability of this assessment mode, particularly at an item level. These threats to validity must be carefully considered by test designs to ensure CBA is used effectively in primary school mathematics classrooms.