Potentials and pitfalls of using ChatGPT in qualitative content analysis – developing a students’ guide

Introduction:

The acceptance of using artificial intelligence (AI) in the qualitative research process varies greatly between researchers. Nevertheless, there is wide agreement in the scientific community that rigor human validation is essential when using AI for (preliminary) coding. Students still have to develop the methodological competences to be able to validate the results provided by AI.

Goals and Methods:

The present project was meant to support qualitative thinking in students and to sensitize them for the potentials, pitfalls and ethics of using ChatGPT. Thirteen nursing management students were trained in using Mayring's qualitative content analysis (QCA). In small groups they were guided to apply inductive and deductive procedures in the analysis of fictional interviews on moral distress in nursing. The categories developed by the six student groups were compared and a final category system was agreed upon. Then, students were guided to develop prompts and compare the results of ChatGPT with their own results. Each group documented the results using a documentation form.

Results:

Regarding to deductive category application, student groups found that ChatGPT e.g. skipped pages or changed category definitions. Concerning inductive category development ChatGPT performed even worse (hallucinated quotes, wrong line numbering, skipped pages, misinterpretations). Students agreed that prompt development is time consuming and still no guarantee that ChatGPT consequently follows the coding rules making the constant human validation laborious.

Learning outcomes were evaluated in terms of attitudes and behavioral intentions to using AI and self-efficacy in QCA (pre-post-test using a 13-item questionnaire) indicating a positive effect of this project.

Conclusions:

Students discussed ethically sound ways of using AI in QC, e.g. treating AI as a source of triangulation or like a second coder, using it to rephrase own codes or to get inspirations for additional codes. These suggestions were summarized as a guideline (living document) for students.