ERIC Number: EJ1263420
Record Type: Journal
Publication Date: 2020
Pages: 20
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-1543-4303
EISSN: N/A
Available Date: N/A
Using an Innovative Standard-Setting Approach to Align Integrated and Independent Writing Tasks to the CEFR
Harsch, Claudia; Kanistra, Voula Paraskevi
Language Assessment Quarterly, v17 n3 p262-281 2020
We report on a standard-setting project in which the Item-Descriptor-Matching Method (IDM) and a complementary benchmarking approach were employed to align a suite of English language proficiency exams to the "Common European Framework of Reference" (CEFR), with a particular focus on the integrated and independent writing exams. Judges' ratings on eight writing tasks and 48 test taker scripts were collected online via SmartSurvey. The judges gave CEFR-level jkudgements for tasks and scripts, they stated which CEFR descriptors they matched against the task demands and scripts, and they evaluated the combined approach and outcomes. Thus, it was possible to monitor how judges applied and interpreted the CEFR descriptors, a prerequisite for establishing alignment validity. Analyses of judgement consistency revealed a high level of consistency on the task judgements and overall performance ratings, yet they also revealed some variations in the selected CEFR descriptors underpinning the judgements. Making these variations transparent facilitated a targeted discussion with an explicit focus on the CEFR, i.e. the framework to which the tests were to be aligned. Overall, the judges reported confidence in using the combined approaches, in their judgements and in the recommended CEFR cut-scores, thus corroborating procedural validity.
Descriptors: Standard Setting, Guidelines, Rating Scales, Definitions, Decision Making, Writing Evaluation, Second Language Learning, Foreign Countries, Benchmarking, Test Items, Language Tests, Task Analysis, Cutting Scores, Validity, Specialists, English (Second Language), Item Response Theory
Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: European Union; United Kingdom (London)
Grant or Contract Numbers: N/A
Author Affiliations: N/A