ERIC Number: ED671451
Record Type: Non-Journal
Publication Date: 2020-Oct
Pages: 33
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Available Date: 0000-00-00
Using Semantic Similarity to Assess Adherence and Replicability of Intervention Delivery. EdWorkingPaper No. 20-312
Kylie L. Anglin; Vivian C. Wong
Annenberg Institute for School Reform at Brown University
Researchers are rarely satisfied to learn only whether an intervention works, they also want to understand why and under what circumstances interventions produce their intended effects. These questions have led to increasing calls for implementation research to be included in high quality studies with strong causal claims. Of critical importance is determining whether an intervention can be delivered with adherence to a standardized protocol, and the extent to which an intervention protocol can be replicated across sessions, sites, and studies. When an intervention protocol is highly standardized and delivered through verbal interactions with participants, a set of natural language processing (NLP) techniques termed semantic similarity can be used to provide quantitative summary measures of how closely intervention sessions adhere to a standardized protocol, as well as how consistently the protocol is replicated across sessions. Given the intense methodological, budgetary and logistical challenges for conducting implementation research, semantic similarity approaches have the benefit of being low-cost, scalable, and context agnostic for use. In this paper, we demonstrate how semantic similarity approaches may be utilized in an experimental evaluation of a coaching protocol on teacher pedagogical skills in a simulated classroom environment. We discuss strengths and limitations of the approach, and the most appropriate contexts for applying this method.
Descriptors: Intervention, Artificial Intelligence, Natural Language Processing, Semantics, Research Problems, Coaching (Performance), Simulated Environment, Replication (Evaluation), Proximity, Language Usage, Preservice Teacher Education, Standards
Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Publication Type: Reports - Research
Education Level: Higher Education; Postsecondary Education
Audience: N/A
Language: English
Sponsor: Institute of Education Sciences (ED)
Authoring Institution: Annenberg Institute for School Reform at Brown University
IES Funded: Yes
Grant or Contract Numbers: R305B140026; R305D190043
Department of Education Funded: Yes
Author Affiliations: N/A