Students' success in physics problem-solving extends beyond conceptual knowledge of physics, relying significantly on their mathematics skills. Understanding the specific contributions of different mathematics skills to physics problem-solving can offer valuable insights for enhancing physics education. Yet such studies are rare, particularly at the high school level. This study addresses the underexplored area of this topic in secondary education by investigating the associations between physics problem-solving performance using a robust methodological framework. We applied exploratory factor analysis (EFA) to identify latent sub-mathematics skills relevant to physics problem-solving and employed structural equation modeling (SEM) to examine the causal impact of these skills on students' performance in physics. The study analyzed data from a municipal-wide assessment involving 1,878 grade 12 students in Southern China. The results demonstrate that mathematics skills impacting high school students' physics problem-solving performance can be categorized into two sub skills, algebraic skills and geometric skills. It also indicates that algebraic skills have a stronger direct effect on physics problem-solving performance compared to geometric skills in high school setting. These findings suggest that integrating focused algebraic training within physics education can significantly improve student outcomes in STEM fields. We recommend that educators design curricula and instructional strategies that emphasize the development of algebraic skills necessary for solving complex physics problems. Additionally, these findings have important implications for policymakers, who should consider integrating targeted mathematics training within physics curricula to foster interdisciplinary learning and better prepare students for challenges in STEM education.