Generative artificial intelligence for project evaluation: comparing large language models and human experts

Abstract

Peer review is at the heart of quality control in research and innovation (R&I) project proposals. Yet, the process remains slow, time-consuming, and increasingly strained by rising academic workloads. This study aimed to investigate the capabilities of Generative Pretrained Transformers (GPTs), a form of artificial intelligence (AI), compared to human experts in the context of R&I project evaluation. The research compares the anonymous scoring of three project proposals by human reviewers and GPTs. Further human experts then evaluated the quality of these reviews. The GPTs demonstrated the same ability to generate comprehensive reviews, with their quantitative scores equivalent to the human averages. Moreover, no substantial differences could be observed in the quality of human and AI reviews. While encouraging, current important limitations of this approach include eventual biases present in the large language model training data. Thus, human judgment is still necessary to ensure accuracy and safety, indicating that AI cannot fully replace humans. In summary, our findings elucidate the combined efficacy of AI and human proficiency, promoting a cooperative framework wherein AI functions as an enhancement tool, consequently maximizing human effectiveness in the evaluation of project proposals.

Ähnliche Arbeiten