This study aims to create and validate an evaluation tool that assesses the learning effectiveness of extended reality applications (e.g., augmented reality, virtual reality, mixed reality, and the metaverse). Six research questions were formulated to guide this study. The Extended Reality in Education (XREd) questionnaire consists of 50-items derived from 25 existing and validated questionnaires and surveys. The tool uses a 1-5 Likert scale and its items correspond to 15 factors, which are categorized into 5 dimensions. Both the factors and dimensions are supported by related theoretical aspects and background, and are grouped as follows: (1) System usability: Ease of use and Instructional clarity, (2) Learner's basic psychological needs: Enjoyment, Autonomy, Self-efficacy, and Social interaction, (3) Learner's learning state: Flow, Presence, and Immersion, (4) Learner's learning performance: Learning outcomes, Learning motivation, and Learning engagement, (5) Learner's learning perceptions: Perceived usefulness, Learning preferences alignment, and Behavioral intention. To validate the tool, descriptive statistics and reliability tests, validity tests, and factor analyses, Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), and Structural Equation Modelling (SEM) were used. An extended reality application was developed and applied in higher education settings over the course of a semester. The experiment data was used to validate the created evaluation tool, which was then used to evaluate the effectiveness of the developed XR application and to answer the 6 research questions set to be explored. The results confirmed the validity of the XREd questionnaire and its effectiveness in evaluating extended reality applications. Additionally, its potential to be modularly integrated to evaluate non-extended reality learning applications was highlighted. Finally, the developed application was positively assessed by students, who regarded it as an effective learning tool that can support and enrich their learning experience and improve learning outcomes.