ERIC Number: ED611395
Record Type: Non-Journal
Publication Date: 2020
Pages: 61
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: N/A
Classification Accuracy and Efficiency of Writing Screening Using Automated Essay Scoring
Wilson, Joshua; Rodrigues, Jessica
Grantee Submission
The present study leveraged advances in automated essay scoring (AES) technology to explore a proof of concept for a writing screener using the "Project Essay Grade" (PEG) program. First, the study investigated the extent to which an AES-scored multi-prompt writing screener accurately classified students as at risk of failing a Common Core-aligned English language arts state test. Second, the study explored whether a similar level of classification accuracy could be achieved with a more efficient form of the AES-screener with fewer writing prompts. Third, the classification accuracy of the AES-scored screeners was compared to that of screeners scored for word count. Students in Grades 3-5 (n = 185, 167, and 187, respectively) composed six essays in response to multiple writing-prompt screeners on six different randomly assigned topics, two essays in each of three different genres (narrative, informative, and persuasive). Receiver operating characteristic (ROC) curve analysis was used to assess classification accuracy and to identify multiple cut scores with associated sensitivity and specificity values, and positive and negative posttest probabilities. Results indicated that the AES-scored multi-prompt screener and screeners with fewer prompts yield acceptable classification accuracy, are efficient, and are more accurate than screeners scored for word count. Overall, results illustrate the viability of writing screening using AES. [This paper was published in "Journal of School Psychology" v82 p123-140 2020.]
Publication Type: Reports - Research
Education Level: Elementary Education; Early Childhood Education; Grade 3; Primary Education; Grade 4; Intermediate Grades; Grade 5; Middle Schools
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R305H170046
Author Affiliations: N/A