ERIC Number: EJ1365721
Record Type: Journal
Publication Date: 2023-Feb
Pages: 29
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0049-1241
EISSN: EISSN-1552-8294
Available Date: N/A
Measuring and Visualizing Coders' Reliability: New Approaches and Guidelines from Experimental Data
Sociological Methods & Research, v52 n1 p525-553 Feb 2023
This study investigates inter- and intracoder reliability, proposing a new approach based on social network analysis (SNA) and exponential random graph models (ERGM). During a recent exit poll, the responses of voters to two open-ended questions were recorded. A coding experiment was conducted where a group of coders coded a sample of text segments. Analyzing the data, we show that the proposed SNA/ERGM method extends significantly our analytical leverage, beyond what popular tools such as Krippendorff's [alpha] and Fleiss's K have to offer. The reliability of coding for individual coders differed significantly for the two questions although they were very similar and the same codebook was used. We conclude that the main advantages of the proposed SNA/ERGM method are the intuitive visualizations and the nuanced measurements. Detailed guidelines are provided for practitioners who would like to use the proposed method in operational settings.
Descriptors: Interrater Reliability, Coding, Social Networks, Network Analysis, Graphs, Models, Visualization, Measurement, Guidelines
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub-com.bibliotheek.ehb.be
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A
Author Affiliations: N/A