The OECD's Programme of International Student Assessment known as PISA, including the Gender Report published in 2015, is a well-received project and its findings keep being transferred into educational policy development. Its quantitative methodology and the sheer volume of data gathered from around 5,100,000 students from 65 countries seem to guarantee scientific rigour and hence, credibility. In this article, we critically examine six topics: (1) The "gender concept" the authors employ, (2) implicit "hypotheses of compensation" (historical, internal, and external), (3) descriptive and correlational statistics and its usage for causal explanations, (4) questions of "external validity," (5) the concepts of "stereotypes" and "role models" and (6) the authors' concept of "innate sex differences." We will show that the PISA Gender Report is based not on well-defined notions of "sex" and "gender," but on what might be called an 'intuitive' gender concept, thus leaving room for theoretically unfounded and mostly implicit hypotheses and subsequent "ad hoc" conclusions. This has led to an interpretation of the findings that sustains a gender dichotomy and reinforces stereotypes of the helpless, social female on the one hand and the self-orienting, natural male on the other.