Measuring Accuracy in AI-Generated Definitions: A Comparison Among Select GPTs Using Cosine Similarity Index

Abstract

<title>Abstract</title> Information generation are highly taking place now-a-days using generative pre-trained transformer (GPT). GPT is widely used in search engines, which generates texts, based on instructions given by humans. ChatGPT, Gemini, Pi and Perplexity are some applications of natural language processing which are widely based on GPT. It is a pre-trained model that uses artificial neural networks to generate the texts in a pattern. These programs are widely used in vast disciplines, most widely in academics. Hence it is imperative to understand the authenticity of the generated texts. The present study uses cosine similarity index to understand the similarity of the texts generated using ChatGPT, Gemini, Perplexity and Pi, along with that of the ground truth. Definitions of some science and social science subjects were generated using the GPT tools used in the present study and the true values were taken as definition provided in the Britannica encyclopedia. The present study opens the avenues for comparing the discipline wise knowledge of these GPT based software using a broader set of keywords.

Ähnliche Arbeiten