Critical thinking is a combination of complex cognitive skills. It is important for the successful acquisition of disciplinary skills in higher education and thus, it is a valuable competency for a new student. The complex nature of critical thinking leads to challenges for its assessment even in performance assessments such as CLA + International (Collegiate Learning Assessment). The aim with this study is to examine internal associations of a critical thinking assessment for new students in higher education. The sample consisted of 1469 first-year students in 18 higher education institutions in Finland. An open-ended performance task and multiple-choice tasks were used to assess six measures of critical thinking, namely analysis and problem solving, writing effectiveness, writing mechanics, scientific and quantitative reasoning, critical reading and critiquing an argument. Exploratory and confirmatory factor analyses were used to analyse the data. A latent component indicated by the measures derived from the performance task was identified. The measures derived from the multiple-choice tasks did not form a factorial structure. Multiple-choice questions are problematic in critical thinking assessment as they focus on individual skills instead of holistic use of skills. Implications for assessment development and higher education are discussed.