This study aimed at evaluating how students perceive the linguistic quality and scientific accuracy of ChatGPT responses to physics comprehension questions. A total of 102 first- and second-year physics students were confronted with three questions of progressing difficulty from introductory mechanics (rolling motion, waves, and fluid dynamics).…

Automated assessment using Natural Language Processing (NLP) has the potential to make English speaking assessments more reliable, authentic, and accessible. Yet without careful examination, NLP may exacerbate social prejudices based on gender or native language (L1). Current NLP-based assessments are prone to such biases, yet research and…

This research aimed to develop a valid and reliable test to be used to detect sixth grade students' misconceptions and errors regarding the subject of fractions. A misconception diagnostic test has been developed that includes the concept of fractions, different representations of fractions, ordering and comparing fractions, equivalence of…

Large-scale standardized tests are regularly used to measure student achievement overall and for student subgroups. These uses assume tests provide comparable measures of outcomes across student subgroups, but prior research suggests score comparisons across gender groups may be complicated by the type of test items used. This paper presents…

Reliability is usually estimated for a total score, but it can also be estimated for item scores. Item-score reliability can be useful to assess the repeatability of an individual item score in a group. Three methods to estimate item-score reliability are discussed, known as method MS, method [lambda][subscript 6], and method CA. The item-score…

In this study, we examined the results and interpretations produced from two different IRTree models--one using paths consisting of only dichotomous decisions, and one using paths consisting of both dichotomous and polytomous decisions. We used data from two versions of an impulsivity measure. In the first version, all the response options had…

International surveys are increasingly being used to understand nonacademic outcomes like math and science motivation, and to inform education policy changes within countries. Such instruments assume that the measure works consistently across countries, ethnicities, and languages--that is, they assume measurement invariance. While studies have…

Recent research provides evidence that an intersectional approach to defining reference and focal groups results in a higher percentage of comparisons flagged for potential DIF. The study presented here examined the generalizability of this pattern across methods for examining DIF. While the level of DIF detection differed among the four methods…

The article is devoted to the problem of survey items that ask for sensitive information. This factor has a significant impact on the quality and comparability of data from international surveys. We propose a methodology that can be used to comparatively study the level of sensitivity of questions. It is often used in public opinion polls as well…

Scores estimated from multidimensional item response theory (IRT) models are not necessarily comparable across dimensions. In this article, the concept of aligned dimensions is formalized in the context of Rasch models, and two methods are described--delta dimensional alignment (DDA) and logistic regression alignment (LRA)--to transform estimated…

This research aims at comparing item characteristics of instruments for assessing the level of mastery in scientific method for elementary students as they were analyzed using Classical Test Theory (CTT) and Item Response Theory (IRT). The two analyses are usually done separately, for difference object, in this moment it was analyzed…

A log-linear model (LLM) is a well-known statistical method to examine the relationship among categorical variables. This study investigated the performance of LLM in detecting differential item functioning (DIF) for polytomously scored items via simulations where various sample sizes, ability mean differences (impact), and DIF types were…

Recruiting participants through gatekeepers has been widely discussed in qualitative research. However, when a sampling frame is not available, surveying through gatekeepers can also be important for quantitative studies. We used three sampling methods to survey guardians of University students: (a) a gatekeeper variant of the time-space sampling,…

Spreadsheets are a core computational tool for practicing engineers and engineering students. While Microsoft Excel, Google Sheets, and other spreadsheet tools have some differences, numerous formulas, functions, and other tasks are common across versions and platforms. Building upon learning science frameworks showing that interactive activities…

The relevance of academic self-efficacy for educational outcomes is well documented. Pre-academic self-efficacy has hardly been studied, and only one study was found to include an assessment of the measurement invariance of the scale used. The aims were to validate the Pre-Academic Learning Self-Efficacy (PAL-SE) scale in a non-university higher…