At the University of Illinois Urbana-Champaign, engineering academics have begun investigating the capabilities of artificial intelligence (AI) tools in academic settings, specifically assessing whether AI can independently complete course work at a satisfactory level. This initiative emerged amidst growing concerns about students using AI models to do their assignments, prompting reconsideration of teaching methods within higher education.
Melkior Ornik, assistant professor in the Department of Aerospace Engineering, together with PhD student Gokul Puthumanaillam, conducted a pilot study in a third-year undergraduate course focused on the mathematics of autonomous systems. Their research tested the performance of ChatGPT, a widely-used generative AI model, on course assignments and exams. As Ornik explained, "What we said is, 'Okay let's assume that indeed the students are, or at least some students are, trying to get an amazing grade or trying to get an A without any knowledge whatsoever. Could they do that?'"
The results, which are detailed in the preprint paper titled "The Lazy Student's Dream: ChatGPT Passing an Engineering Course on Its Own," revealed that the AI was able to achieve the equivalent of a low B grade in the course using the simplest free version of ChatGPT, based on the GPT-4 model.
Performance varied considerably depending on the nature of the tasks. ChatGPT excelled in closed-form problems such as multiple choice questions or straightforward calculations, achieving near-perfect scores. However, in tasks requiring more complex reasoning and problem-solving — including open-ended design projects that necessitate proposing solutions, detailed explanations, and critical analysis supported by graphs — the AI struggled and garnered a grade equivalent to a D. Ornik noted, "there was still a significant disparity between the different types of problems that [ChatGPT] could deal with or it couldn't deal with."
Based on these findings, Ornik advocates for a re-examination of educational approaches in the era of accessible generative AI. Drawing a parallel with the historical introduction of calculators in classrooms, he remarked, "Before calculators, people would do these trigonometric functions... then... the calculator... was maybe not perfect but decently competent. So we said, 'okay well I guess we'll trust this machine.'"
The central question he poses is about the value and content of instruction: "Is it that we should continue teaching the same stuff that we do now, even though it is solvable by AI, just because it is good for the students' cognitive health? Or is it that we should give up on some parts of this and we should instead focus on these high-level questions that might not be immediately solvable using AI? And I'm not sure that there's currently a consensus on that question."
Ornik has engaged with colleagues from the university's College of Education regarding the rationale behind teaching mental arithmetic and memorisation in early education stages, despite the widespread availability of calculators and smartphones. He shared that the consensus is these practices support cognitive development and future learning capabilities. He suggests this type of discourse should be extended to higher education curricula in light of AI’s growing presence.
In terms of strategies, Ornik outlined three possible responses to the presence of AI in academic work. The first is to treat AI as an adversary, imposing exam formats such as oral exams and custom assignments to minimise the possibility of AI usage. The second is to embrace AI as a collaborative tool, teaching students to integrate it effectively into their studies. The third approach, which Ornik favours, is to acknowledge AI as an existing resource and focus on teaching students critical evaluation skills to ensure responsible and discerning use. He emphasised that students often tend to over-rely on computational tools and that education should teach them to verify AI-generated information, saying, "you should use AI when it makes sense but you should also be sure that whatever it tells you is correct."
While AI is pervasive in discussions about the future of technology, Ornik noted uncertainties surrounding its sustainability, comparing the current climate to the dot-com bubble of the early 2000s. He observed, "I was looking at barbecue grills – the barbecue is AI powered. I don't know what that really means... They just call it AI." Additionally, he highlighted ongoing issues such as data privacy and copyright, which have yet to be fully resolved.
Looking ahead, Ornik and his colleagues plan to expand their research to encompass multiple engineering courses, aiming to develop educational modules that introduce students to the capabilities and limitations of AI like ChatGPT. One such initiative includes creating a critical thinking module that would inform students about AI’s strengths and its potential to make significant errors, especially in relation to course content.
Furthermore, the team intends to experiment with adapting course assessments and teaching materials to the reality of generative AI. Ornik suggested that while some course content might remain valuable with adjusted evaluations, other content might warrant reconsideration regarding its continued relevance.
This study indicates a shifting landscape in higher education, as institutions seek effective ways to integrate AI tools into their pedagogical frameworks while maintaining academic integrity and fostering critical thinking skills among students.
Source: Noah Wire Services