Expert judgement has been frequently employed with reading assessments to gauge the skills potentially measured by test tasks, for purposes such as construct validation or producing diagnostic information. Despite the critical role it plays in such endeavours, few studies have triangulated its results with other types of data such as reported…

Evolutionary trees are central to learning about evolutionary processes, yet students at all educational levels struggle to read and interpret them. The synthetic tree-reading model (STREAM), based on published and not yet empirically tested models, was tested to determine whether the assumed hierarchy of the model could be substantiated and how…

Although an accurate measure of vocabulary size is integral to understanding the proficiency of language learners, the validity of multiple-choice (M/C) vocabulary tests to determine this has been questioned due to users guessing correct answers which inflates scores. In this paper the nature of guessing and partial knowledge used when taking the…

Using a multiphase approach, the purpose of the present study was to develop a psychometrically sound questionnaire to measure protégés' perceptions of peer athlete mentoring functions. Phase 1 consisted of three stages: (a) item development, (b) assessment of content validity via think-aloud interviews with peer mentored athletes, and (c)…

In line with the widespread proliferation of digital technology in everyday life, many countries are now beginning to use computer-based exams (CBEs) in their post-primary education systems. To ensure that these CBEs are delivered in a manner that preserves their fairness, validity, utility and credibility, several factors pertaining to their…

Artificial intelligence (AI) technologies have the potential to reduce the workload for the second language (L2) teachers and test developers. We propose two AI distractor-generating methods for creating Chinese vocabulary items: semantic similarity and visual similarity. Semantic similarity refers to antonyms and synonyms, while visual similarity…

Background: The development of a set of questions is a central element of examination development, with the validity of an examination resting to a large extent on the quality of the questions that it comprises. This paper reports on the methods and findings of a project that explores how educational examination question writers engage in the…

There is a consensus that the items proposed by the Program for International Student Assessment (PISA) program allow us to focus on the outcomes of the processes of appropriation and transformation of learning tools at the end of compulsory schooling, particularly regarding the key competencies for lifelong learning and citizenship in digital…

This article examines the test-taking strategies of high- and low-scoring Chinese-speaking participants when they answer English multiple-choice reading comprehension questions. Thirty-two participants took a TOEIC reading test, provided think-aloud protocols, and joined a post-task interview. The data come primarily from qualitative analysis and…

We report on the development, validation, and implementation of a collection of test items designed to detect misconceptions related to first-year computer science courses. To this end, we reworked the development scheme proposed by Almstrum et al. ("SIGCSE Bulletin" 38(4):132-145, 2006) to include students' artifacts and to…

The goal of this study was to investigate the usefulness of person-fit analysis in validating student score inferences in a cognitive diagnostic assessment. In this study, a two-stage procedure was used to evaluate person fit for a diagnostic test in the domain of statistical hypothesis testing. In the first stage, the person-fit statistic, the…

Pondering a math problem while she swings her sneakered feet from a chair, 12-year-old Andrea Guevara is helping researchers design an assessment that will shape the learning of 19 million students. The 8th grader, who came to the United States from Ecuador three years ago, is trying out two ways of providing English-language support on a…

In Germany, science education standards for students at the end of grade nine have been in existance since 2005. Some of these standards are dedicated to scientific inquiry (e.g. experimentation). They describe which abilities learners are expected to possess at the end of grade nine. In the USA, several documents describe standards for…

Even if national and international assessments are designed to be comparable, subsequent psychometric analyses often reveal differential item functioning (DIF). Central to achieving comparability is to examine the presence of DIF, and if DIF is found, to investigate its sources to ensure differentially functioning items that do not lead to bias.…

The domination of the information processing approach has shifted research from problem solving strategies to the structure and organization of knowledge that characterizes expertise. The purpose of this study was to compare the reasoning processes of 12 clinicians and 40 medical students as they responded to 6 positively stated multiple choice…