The Students Who Used ChatGPT Scored 57.5%. The Ones Who Didn't Scored 68.5% — On a Test Nobody Warned Them About.

Forty-five days after the assignment was over, a researcher in Rio walked back into the room with a test nobody was expecting.

André Barcaui recruited 120 undergraduate business administration students, randomly split them into two groups of 60, and gave both the same assignment: research basic artificial intelligence concepts, including ethics, societal impacts and technical foundations. Half were allowed to use ChatGPT. Half weren't. The AI group worked with GPT-4's free web interface; the control group had textbooks, articles, library databases and ordinary web search. Both prepared a 10-minute presentation.

Then came the surprise. Forty-five days later, 85 of them sat a twenty-question multiple-choice exam built to test conceptual understanding rather than memorized definitions.

The ChatGPT group got 57.5% correct. The traditional group got 68.5%. t(83) = −3.19, p = .002, Cohen's d = 0.68. That effect size is not a rounding error. It's the difference between a student who can use what you taught them and one who can't.

"This suggests that unrestricted ChatGPT use impaired long-term retention, likely by reducing the cognitive effort that supports durable memory," Barcaui writes in the paper, published in Social Sciences & Humanities Open.

The part that should worry course creators

The AI group wasn't lazy. They were efficient. Barcaui tracked preparation time and the chatbot let students finish much faster. Faster, and then emptier.

Prior experience with chatbots made no difference either. Frequent users scored just as badly as newcomers, which Barcaui reads as an illusion of competence: the software makes learners feel like they understand the material better than they do.

Now map that onto the thing you sell. A course business doesn't get paid for delivery. It gets paid for outcomes, which turn into testimonials, renewals, referrals and the case studies on your sales page. An AI assistant bolted into your program optimizes the one metric that doesn't pay you — speed of completion — while quietly deleting the struggle that produces the result you advertise.

Where to protect the friction

Not everywhere. Friction in onboarding kills activation. Friction in the core skill is the product.

Keep AI out of first attempts. Barcaui's own suggestion for future research is testing a sequence where students try to solve a problem alone before consulting a chatbot for feedback. That ordering is free to implement and is the single easiest curriculum change here.

Use AI as critic, not author. Let it mark the student's work. Don't let it produce the work.

Delete it from retrieval. Quizzes, recall prompts, week-four check-ins. If the bot is open, nothing is being retrieved from memory.

Automate the boring layer. Scheduling, transcripts, admin, summaries of things nobody needs to remember.

The caveats are real and Barcaui lists them. Nearly thirty percent of the original participants never returned for the test, and the sample was business students at a single Brazilian university. One trial, one cohort, one subject.

But the direction of the finding runs exactly opposite to the direction the industry is sprinting. Every platform roadmap this year has an AI tutor on it. Very few have a plan for what happens to the student forty-five days later.