Creativity in science education and research on scientific creativity are becoming increasingly significant. Several domain-specific assessment tools focusing on different aspects of scientific creativity have been developed. Some of these tools consider a wide range of aspects, while others focus on very specific science-related aspects, thereby lacking an explicit focus on problem solving. Therefore, the authors developed the novel divergent problem solving ability in science test (DPAS) to assess students' potential to design different possible solutions for a particular scientific problem. The test was created as an outcome measure in intervention studies. It consists of two subtests and 13 subject-specific items. This study aims to broadly assess the construct validity and reliability of the DPAS and investigate the factorial longitudinal invariance and group invariance (group receiving interventions aimed at fostering divergent problem solving vs. control group) of the DPAS. Using data from 1,106 secondary school students from Austria, we found support for the construct validity and reliability of the DPAS scores. Moreover, we established robust longitudinal and group invariance of the DPAS, demonstrating its suitability for intervention studies.