Is ChatGPT smart enough to get you through college? Maybe, but at what cost? A new study from the University of Illinois Grainger College of Engineering reveals that while the free version of ChatGPT can ace straightforward math homework, it struggles with higher-level reasoning, potentially leading to a passing grade with a significant knowledge gap.
Researchers put ChatGPT through a semester-long undergraduate control systems course and compared its performance to human students. The results were surprising: ChatGPT earned an "A" on structured, math-based assignments but received a "D" on open-ended problems demanding critical thinking. This brought its overall grade down to a low B (82%), while the human students achieved an average of 84.85%.
"We found ChatGPT technology can get an A on structured, straightforward questions. On open-ended questions it got a 62, bringing ChatGPT's semester grade down to an 82, a low B. The class average for the human students was 84.85 percent because they could handle the problems that required higher-level reasoning," said Ph.D. student Gokul Puthumanaillam.
The study highlights a crucial concern: a student relying solely on ChatGPT could pass a course without truly grasping the material. They might excel in simple math but fail in analysis, ultimately hindering their learning.
So, what does this mean for the future of education? Melkior Ornik, Puthumanaillam's advisor, believes educators need to adapt. "Like calculators in math classes, ChatGPT is a tool that's here to stay and that students will use. What the results of this study pointed out to me is that I need to adjust as an educator," said Puthumanaillam's advisor Melkior Ornik. "I plan to consider how I design my courses so that, over time, I include more higher-level questions, perhaps including project-based assignments. Students will still use programs like ChatGPT to do the simpler math problems, but by adding more open-ended questions, they'll also reach a higher level of critical thinking and truly learn the material."
This could involve incorporating more open-ended questions and project-based assignments to encourage critical thinking and deeper understanding. While ChatGPT can quickly solve problems, its accuracy can be questionable, and it sometimes hallucinates information, even when provided with all the necessary course materials.
For example, the study found ChatGPT using technical jargon like "quasi periodic oscillations" that were never used in class.
The researchers opted to use the free version of ChatGPT, reasoning that the average student wouldn't pay for the premium version. However, they noted that the premium version might perform slightly better on analytical questions and handle more complex problems due to its larger memory capacity.
Interestingly, ChatGPT did show some ability to learn from its mistakes. When corrected on a multiple-choice question, it performed better on a variation of the same question. However, its overall progress was limited, with its final score remaining relatively stagnant.
The research underscores the importance of adapting educational strategies to leverage AI as a tool while ensuring students develop essential critical thinking and analytical skills. ChatGPT, while helpful for simple tasks, cannot replace the human capacity for reasoning and comprehension that is vital for true learning. The key is to adjust teaching methods to encourage higher-level thinking and project-based assignments.