ERIC Number: EJ1012391
Record Type: Journal
Publication Date: 2013
Pages: 14
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0267-1522
EISSN: N/A
Available Date: N/A
Problems in Estimating Composite Reliability of "Unitised" Assessments
Bramley, Tom; Dhawan, Vikas
Research Papers in Education, v28 n1 p43-56 2013
This paper discusses the issues involved in calculating indices of composite reliability for "modular" or "unitised" assessments of the kind used in GCSEs, AS and A level examinations in England. The increasingly widespread use of on-screen marking has meant that the item-level data required for calculating indices of reliability is now routinely available for most (but not all) units of unitised assessments. Whilst it is relatively straightforward to obtain indices of reliability at unit level, it is far more complex to obtain indices at overall assessment level because of problems created by: (i) the number of different possible "routes" to the final assessment; (ii) the different knowledge, skills and understanding assessed in different units; (iii) the wide variety in item type and size within and across units; (iv) the fact that the item-level data required for calculating reliability indices is not available (or does not exist) for certain units and (v) the different intended weighting of different units in the composite total and the possible distortion of these weights by use of the Uniform Mark Scale. We derive and compare several indices of composite reliability for, as an example, a three-unit assessment using both classical test theory and item response theory. We conclude that while it is desirable to derive and report indices of reliability at unit level, it is less appropriate at the level of the whole assessment. (Contains 6 tables, 1 figure, and 2 notes.)
Descriptors: Foreign Countries, Exit Examinations, Secondary Education, Test Reliability, Educational Assessment, Item Response Theory, Test Theory, Computation, Achievement Tests, Error of Measurement
Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Publication Type: Journal Articles; Reports - Evaluative
Education Level: Secondary Education
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Identifiers - Location: United Kingdom (England)
Grant or Contract Numbers: N/A
Author Affiliations: N/A