ERIC Number: ED671619
Record Type: Non-Journal
Publication Date: 2024-Jul
Pages: 57
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Evaluating the Correspondence between Expert Visual Analysis and Quantitative Methods
Grantee Submission
Visual analysis is the primary methodology used to determine treatment effects from graphed single-case design data. Previous studies have demonstrated mixed findings related to interrater agreement between both expert and novice visual analysts, which represents a critical limitation of visual analysis and supports calls for also presenting statistical analyses (i.e., measures of effect size). However, few single-case design studies include results of both visual and quantitative analyses for the same set of data. The current study investigated whether blind review of single-case graphs constructed using up-to-date recommendations by experts in visual analysis would demonstrate adequate interrater agreement and have correspondence with an effect size metric, log response ratio (LRR). Eleven experts (i.e., professors in school psychology and special education with visual analysis experience) evaluated 26 multiple baseline graphs evaluating implementation planning, a fidelity support, on educators' implementation and student outcomes, presented in a standardized format without indication of the variable being measured. Results suggest that there was strong correspondence between raters in their judgments of the presence or absence of treatment effects and meaningfulness of effects (particularly for graphs of adherence and quality). Additionally, a quadratic relationship was observed between aggregate results of expert visual analysis and effect size statistics. Implications for future research and limitations are discussed. [This paper was published in "School Psychology" v39 n6 p557-571 2024.]
Publication Type: Reports - Research; Information Analyses
Education Level: N/A
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: N/A
IES Funded: Yes
Grant or Contract Numbers: R324A210266
Department of Education Funded: Yes
Author Affiliations: 1Institute for the Collaboration on Health, Intervention, and Policy, University of Connecticut; 2Department of Educational Psychology, University of Connecticut; 3Department of Health and Behavior Studies, Teachers College, Columbia University; 4Department of School Psychology, University of California, Riverside