Evaluating <scp>ChatGPT</scp> as an educational resource for patients with multiple myeloma: <scp>A</scp> preliminary investigation

Abstract

The findings of this study highlight a 95% accuracy rate in ChatGPT responses, as assessed by five myeloma specialists, underscoring its potential as a reliable educational tool. Multiple myeloma (MM) is a complex hematological malignancy characterized by malignant plasma cell proliferation usually within the bone marrow.1 Patients diagnosed with MM face multiple challenges, including understanding the intricacies of their condition, available treatment modalities, side effects, supportive care, and the consequent prognostic implications. For that reason, patient education is essential in ensuring an empowered patient population capable of making informed healthcare decisions. With the advent of artificial intelligence (AI), there is emerging potential to leverage these technologies for patient education. ChatGPT, an AI-based language model, represents one such promising avenue.2 While this AI tool has found application in numerous domains, its potential in various medical fields is being explored.3-5 This study aimed to explore the utility of ChatGPT as an educational resource for MM patients, evaluating its accuracy and safety in disseminating medical information. We developed a comprehensive 21-question questionnaire (Table 1) encompassing a wide range of topics pertinent to MM—from diagnosis to treatment options and prognosis, which served as the prompts for the study conducted during the month of January 2024, utilizing OpenAI's ChatGPT version 3.5.0 to generate responses. ChatGPT's responses were generated based on its training data, and the accuracy of the information provided was meticulously evaluated by five different myeloma specialists actively practicing modern myeloma care in two different states (two from Florida and three from Ohio) in the Cleveland Clinic Health System. The specialists individually and privately assessed each response provided by ChatGPT and categorized them as accurate, inaccurate, or harmful. It is important to note that if any response received a classification of inaccurate or harmful by at least one of the five specialists, it was categorized as such. This categorization process aimed to establish the efficacy and safety of using ChatGPT as an educational resource for MM patients. Our analysis of ChatGPT's responses to the 21-question questionnaire revealed a predominant trend of accuracy, with all responses categorized as accurate based on evaluation by five myeloma specialists, except for Question 3. This question, “What are the available treatment options for MM?,” was flagged by three out of the five MM experts for including an incomplete answer. The omission of information on CAR T-cell therapy and bispecific antibodies was noted. As a result, we obtained a score of 20/21, which corresponds to a 95% accuracy rate. Notably, for Question 2, “What are the common symptoms of MM?,” concerns were raised by one myeloma expert regarding clarity for a non-medical public. Despite these isolated concerns, the overall assessment underscores ChatGPT's potential as a reliable source of information for patients seeking to understand various aspects of their condition, including diagnosis, treatment options, and prognosis. Importantly, none of the responses were categorized as harmful by the specialists. This absence of harmful information further solidifies ChatGPT's safety in disseminating medical information to patients with MM, indicating a low risk of misinformation or potential harm. The study's findings underscore ChatGPT's potential as a valuable educational tool for patients seeking information about MM. The high accuracy rates of its responses emphasize its role in providing accessible medical insights, although certain limitations were observed, primarily regarding response completeness and language complexity.3 This study serves as a stepping stone in leveraging AI's potential in patient education. The identified areas for improvement, such as refining language simplicity and ensuring comprehensive responses, offer valuable insights for further development. Collaborations between healthcare providers and AI developers become pivotal in refining and optimizing ChatGPT's capabilities, addressing its limitations, and tailoring it to better meet the needs of patients with varying levels of medical understanding. The absence of harmful information in ChatGPT's responses to our questions is reassuring but does not eliminate the need for vigilance. Continuous monitoring, refining, and validating the accuracy of AI-generated medical information remain imperative to ensure patient safety and reliable dissemination of medical knowledge.6 Moreover, considering the evolving nature of medical research and treatments, regular updates and integration of the latest evidence-based information into AI models like ChatGPT become essential. This continual learning process ensures that the information provided remains current and aligned with the latest advancements in the field of MM. In conclusion, while ChatGPT shows promise as an educational resource for MM patients, it is essential to recognize its limitations and the indispensable role of human medical expertise. This study lays the groundwork for future collaborations aimed at refining AI tools like ChatGPT, ensuring they complement healthcare providers' efforts and contribute effectively to patient education while maintaining a focus on safety, accuracy, and accessibility. This study delineates a promising potential for ChatGPT as an effective educational resource for MM patients, evidenced by a high accuracy rate in its responses and positive specialist assessments. Nevertheless, the necessity for ongoing refinements to ensure the highest level of accuracy and safety cannot be overstated. As AI continues to evolve, its integration within the patient education paradigm could play a pivotal role in improving patient care and understanding, albeit as a complementary tool to the indispensable human medical expertise. The authors declare no conflicts of interest. The data that supports the findings of this study are available in the supplementary material of this article (Data S1 and S2). Additional data available from the corresponding author upon request. Data S1. Supporting Information. Data S2. Supporting Information. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Ähnliche Arbeiten