Assessing ChatGPT’s Capability to Generate Course Learning Outcomes

Abstract

ChatGPT is transforming higher education through the utilization of advanced neural networks and large language models to produce human-like content. However, its integration in various educational contexts has yet to be comprehensively evaluated. This study aims to assess ChatGPT’s ability to create course learning outcomes (CLOs) effectively, which remains unexplored. ChatGPT was instructed to generate 10-15 CLOs for 10 courses in the information systems department, based on the course title, description, and topics. The CLOs generated by ChatGPT were then evaluated using two different methods: 1) automatic evaluation metrics (BERTScore) to measure semantic similarity and 2) experts who provided a more comprehensive evaluation of the CLOs relevance to the course and diagnose the characteristic of poorly generated CLOs. The findings demonstrate that ChatGPT can serve as a supportive tool for instructors. The majority of the generated CLOs are either similar to or novel additions to those developed by humans. Nonetheless, approximately 20% of experts CLOs were missing and 22% of the produced CLOs were incorrect, mainly identified as irrelevant. Therefore, it is crucial to exercise caution when using this tool and to rely on human expertise to assess the CLOs generated by ChatGPT.

Ähnliche Arbeiten