Evaluating ChatGPT's Efficacy in Qualitative Analysis of Engineering Education Research

Abstract

Abstract This study explores the potential of ChatGPT, a leading-edge language model-based chatbot, in crafting analytic research memos (ARMs) from student interview transcripts for use in qualitative data analysis. With a rising interest in harnessing artificial intelligence (AI) for qualitative research, our study aims to explore ChatGPT's capability to streamline and enhance this process. The research is part of a mixed-methods project examining the relationships between engineering students' team experiences, team disagreements, and engineering identities. Our team had previously developed an interview protocol for collecting qualitative data and initiated analysis using coding methods and ARMs for individual transcripts. We designed an ARM Development Guidelines document to ensure consistency among four team members in the ARM creation process. The guidelines include a set of key questions that each ARM should aim to address. Our objective is to assess ChatGPT's proficiency in creating ARMs based on our development guidelines and compare its outputs with human-written ARMs for accuracy and depth of insight. For this purpose, we selected two student interview transcripts. A structured analysis protocol for ChatGPT was devised in adherence to the ARM Development Guidelines. Two team members, experienced in qualitative analysis and ARM composition, drafted ARMs for the chosen transcripts using the same guidelines, enabling a direct outcome comparison. Subsequently, a rigorous validation process was conducted, using rubrics to assess narratives from both methods. The manual ARM authors performed a self-assessment, while the other researchers conducted a blind evaluation of the human-generated and AI-generated ARMs. We used two rubrics for this comparison. A general rubric gauged accuracy, clarity, analysis time, and usefulness. A specialized rubric was used to determine if the ARMs address the topics laid out in the ARM guidelines, such as self-identification, perceptions of engineering, teamwork descriptions, connections between identity and team experiences, comparisons with other interviews, and reflections. In this paper, we describe our research methodology, present our findings, evaluate the advantages and limitations of ChatGPT in qualitative analysis within engineering education research, and provide guidance for future research directions. We aim to shed light on the capabilities of ChatGPT in qualitative analysis and contribute to the ongoing dialogue on harnessing AI for research in engineering education. Our findings will inform researchers and practitioners about the benefits, challenges, and best practices associated with integrating AI-powered tools such as ChatGPT into qualitative research methods.

Ähnliche Arbeiten