A theoretical framework of consensus in qualitative coding: Methodological reflections from a survey study

Introduction

Intercoder agreement (ICA) is a quantitative measure for determining the extent of reliability in qualitative coding conducted by multiple researchers. Given the subjectivities of coding among different researchers, ICA remains a key indicator of rigorous data analysis for mixed methods and quantitative researchers guarding against bias. However, reducing analytic validity to a simple score overlooks the processes of discussion and reflexivity that foster that agreement in the first place. This prompts methodological inquiries that operationalize ICA not as a final arbiter for research rigor but as a confirmatory indicator of negotiation of consensus among coders.

Goals/methods

To that end, this presentation presents a theoretical framework for identifying the extent of consensus among coders in qualitative research. The presentation discusses the coding and analysis of qualitative data from a survey study of caregivers exploring their views on their children's sleep health. Through analyzing open-ended survey responses to questions regarding community and parenting factors, the presenters documented the extent of collaboration in terms of process and outcome through analytic memos and discussions.

Results

Refinement of this framework is based in part on a calculation of ICA after each iteration of coding to determine the extent of consensus as it evolves over time. The use of intercoder agreement in this manner validates the elements of the framework, which includes processes such as active negotiation as well as dispositions such as reflexivity.

Conclusion

The presentation of this framework has implications for qualitative researchers who aim to privilege the methods for reaching consensus in their data analysis rather than  a strictly quantitative measure of the reliability of outcomes.