Solving word problems involving 'Time' is an important skill but poor mastery of the skill among elementary students has often been reported in the literature. In addition, the available diagnostic tools in the literature might be less efficient for identifying the various errors made by many students in solving word problems. Thus, an online problem solving Cognitive Diagnostic Assessment (CDA) with Ordered Multiple-Choice (OMC) items was developed as a web application with automated scoring features to increase the efficiency in assessing Grade Five students' mastery level of word problem-solving attributes involving the topic of Time. The online problem-solving CDA with OMC items was developed based on the principled assessment design which comprised four building blocks: (i) construct map, (ii) item design, (iii) outcome space, and (iv) measurement model. Newman's Error Analysis was adapted to specify the construct map required for the development of OMC items. In this article, we documented the empirical evidence regarding item quality, the validity of the cognitive models, the reliability of the instrument, and the diagnostic analysis of students' responses. This study involved 128 Grade Five students in three elementary schools. The results indicated that the instrument showed appropriate difficulty level and discrimination power, satisfactory model-data fit, and high reliability, but most of the students were not at the highest mastery level of the word problem-solving attributes. The findings of the study suggested that the web application could be a valid and reliable diagnostic tool for pinpointing students' errors made in solving word problems. Besides, the findings also highlight the need for intervention to enhance the students' mastery of word problem-solving attributes.