Newsletters
People who used an artificial intelligence (AI) chatbot to solve simple math and reading problems solved fewer of them and skipped more once the tool was taken away, according to a peer-reviewed study.
The findings come from three randomized experiments with 1,222 participants, presented at the Conference on Language Modeling (COLM), held Oct. 6-9 in San Francisco, California.
Grace Liu of Carnegie Mellon University in Pittsburgh led the study with co-author Brian Christian of the University of California, Berkeley.
The researchers recruited 1,222 participants for three randomized controlled experiments, each lasting 13 to 15 minutes. Participants were recruited online through the research platform Prolific, and the final analysis included 1,058.
In the first experiment, 354 participants were randomly assigned to a group with access to an AI chatbot or a group without it. Both groups received the same 12 fraction problems.
The AI group could obtain answers immediately or request clues. The researchers then removed the chatbot without warning and gave everyone a final set of three problems.
Participants could skip problems without penalty, so skipping was a deliberate choice not to engage. After wrong answers, participants saw the correct solution.
In the final three problems, the AI group solved 57% and skipped 20%, compared with 74% and 11% for the group that never used the chatbot.
The second experiment repeated the design with 667 participants and added a pretest to screen out people who could not solve the most basic problems. The AI group again solved fewer problems in the final phase, 71% compared with 77%. The skip-rate gap (10% vs. 7%) was not statistically significant across the full sample.
Among AI users in that experiment, 61% said they used the chatbot to get direct answers and 27% said they used it for hints or clarification.
Participants who relied on direct answers solved 65% of the final problems, compared with 77% for the control group, and skipped more.
The authors describe these comparisons between usage groups as cross-sectional and not necessarily causal.
The third experiment tested reading comprehension with 201 participants. Both groups answered five questions before the chatbot was disabled for three more.
The AI group solved 76% and skipped 8% in the final phase, compared with 89% and 1% for the control group.
Liu told Live Science that the most significant finding is that “people who offload more have higher declines in persistence.”
She said the concern relates to what cognitive scientists call “desirable difficulties,” the productive struggle that builds skill over time.
Liu acknowledged that calculators also let people offload computation, but said AI is distinct because it lets people offload reasoning across many types of tasks.
The authors suggest one possible reason for the drop in persistence: people may become used to receiving instant answers and come to see slower methods as inefficient.
The study tested only short sessions, and researchers measured participants immediately after the chatbot was removed. Liu said scientists do not yet know how small instances of offloading affect lasting competence.
Answering that would require tracking the same people over a long period and regularly measuring learning outcomes across a wide range of tasks.
The authors argue that AI chatbots should be designed more like tutors, giving directions toward a solution rather than the solution itself.
In the second experiment, participants who used the chatbot for hints showed no drop in persistence compared with the control group.
In a statement published by UC Berkeley, Christian said work should aim for a future in which human abilities are augmented, not replaced.
Set as preferred source
Set as preferred source
1 min read
1 min read
