A Comparative Analysis of Mental Models in Floating, Suspending, and Sinking Phenomena: Evidence from Students and Large Language Models (ChatGPT and DeepSeek)

Abstract

The increasingly widespread use of Large Language Models (LLMs) in education creates opportunities for further investigation into epistemic reasoning abilities, particularly the characteristics of mental models. Therefore, this study aimed to characterize the mental models of LLMs, specifically the AI models DeepSeek and ChatGPT. The technologies were also compared to students in relation to the phenomena of floating, suspending, and sinking using descriptive-quantitative and a partial descriptive analysis. This study was conducted in a descriptive-quantitative design employing a cross-sectional comparative design. The quantitative analysis compares test scores across all subject groups, while the descriptive analysis is used to characterize students’ and AIs’ mental models. The study subjects consisted of 709 students and 120 chats for two AI models as respondents. Each AI was treated as comprising 60 respondents, with the item–test pair regarded as a distinct, independent chat instance. Based on the data analysis, mental models of students were predominantly categorized as initial, with only a limited decline following an increase in educational level. This decline was accompanied by a similarly limited increase in the proportion of scientific mental models, showing a modest influence of curricular interventions experienced by the students. The AI models exhibited higher proportions of scientific categories, but the mental models showed inconsistency when contexts changed, and the underlying aspects remained the same. Therefore, the reasoning was not classified as global coherence. The students and the AI models exhibited misapplications of density-based reasoning and Archimedes’ principle in several cases or phenomena presented in the test. The results showed that students and AI models have limitations in their mental models.

Ähnliche Arbeiten