Collaborative problem solving (CPS) is an important 21st-century skill that is crucial for both career and academic success. However, developing a large-scale and standardized assessment of CPS that can be administered on a regular basis is very challenging. In this report, we introduce a set of psychometric considerations and a general scoring strategy around assessing CPS, summarized based on the results of the extensive empirical studies we conducted at Educational Testing Service (ETS) over the past 6 years. Using the ETS Collaborative Science Assessment Prototype as an example, we show how these psychometric considerations have been incorporated into the development of the assessment prototype and how the scoring strategy has been implemented.