Confidently Wrong, Together

A new preprint reports that giving people AI advice made them worse at answering questions and, at the same time, far more sure of their answers. If the finding holds, the most important cost of AI-assisted work is not that machines are sometimes wrong. It is that using them appears to strip humans of the ability to notice when they, or the machine, don't know something.

The paper, posted to PsyArXiv on 15 July 2026 by Chiara Marcoccia, Valerio Capraro and Walter Quattrociocchi, is titled, plainly, "AI advice suppresses people's willingness to say 'I don't know', even when the advice is wrong and accuracy is incentivized." ¹ Across five experiments with 3,132 participants, people answered questions with or without advice from an AI assistant — deliberately, a model the researchers chose because it gets these particular questions wrong, so the experiment isolates what the advice does to the human rather than how good the advice is. With AI advice in hand, accuracy fell from 27% to 9%, high-confidence responses rose from 30% to 76%, and the willingness to answer "I don't know" collapsed from 44% to 3%. Paying participants for correct answers helped a little, accuracy in the incentivised AI condition reached 16%, but nowhere near closed the gap.

What the numbers actually say. Read carefully, the study is not a claim that chatbots are wrong three times out of four. It is a claim about what happens to a human being who has consulted one. People consulted a system, walked away with a worse answer than they would have produced alone, and reported themselves more certain than they had been before asking. The authors' own abstract puts it starkly: with AI advice available, people answered more questions, hedged less, and were correct about a third as often as when working alone. ¹

That is a specific kind of harm, and it is worth being precise about it. It is not the harm of misinformation, where a bad claim spreads. It is not the harm of automation, where a job disappears. It is the erosion of a cognitive signal — the internal wobble that tells you when to hedge, when to look something up, when to say you're not sure. That signal is how humans have historically caught their own errors and each other's. In this experiment, it went away.

Who this happens to. The study was a controlled experiment with a general participant pool, and I don't want to overreach on demographics the paper doesn't break out. But it is worth naming the obvious asymmetry in who is currently being handed these tools as a matter of course. Students doing homework. Junior professionals drafting memos. Patients navigating symptoms between appointments. Benefits claimants working out what they're entitled to. Call-centre agents reading AI-suggested replies off a screen.

In each of those settings, the person on the other end of the interaction, the teacher, the senior, the doctor, the caseworker, the customer, is being asked to trust a confidence signal that the study suggests is now roughly decoupled from accuracy. The consequences of confidently-wrong answers do not distribute evenly. They land hardest on the people whose judgement is least likely to be double-checked: the students whose teachers are stretched, the patients whose GPs have eight minutes, the claimants whose caseworkers are working from the same AI summary the claimant is.

The productivity story looks different from here. Much of the current enterprise narrative treats AI assistance as a straight productivity gain: same worker, same task, faster output. The Marcoccia et al. result complicates that arithmetic. If accuracy falls while volume rises, some of what is being measured as productivity is really the substitution of confident wrong answers for slower right ones (or slower "I don't know"s, which are frequently the correct answer). That is not a productivity gain. It is a transfer — from the person or institution that would have caught the error to whoever now has to live with it.

I want to take seriously the objection that this is one preprint, on one model, in one task setting, and that people will get better at using these tools over time. All of that is true. The paper has not been peer-reviewed. Calibration may improve with familiarity. Newer models may hedge more honestly. It also sits alongside a growing body of adjacent work — on cognitive offloading, on the way confident AI outputs suppress users' willingness to disagree — that points in the same direction.

But even granting the caveats, the direction of the finding matters more than its exact magnitude. A drop in "I don't know" from 44% to 3% is not the kind of effect that survives only at the extremes. Something is happening to the epistemic posture of people who consult these systems. If it is even a fraction as large in the wild as it is in the lab, the deployments already in place, in schools, in clinics, in customer service, in government-facing chatbots, are running an experiment on their users that no one has consented to.

What follows from taking this seriously. The design implication is not subtle. If an AI system's outputs strip users of their own uncertainty signal, then the system has to put the uncertainty back. That means outputs that visibly distinguish "confident" from "guessing", that surface disagreement between sources rather than laundering it into a single fluent paragraph, that make "I don't know" a first-class output rather than an embarrassment the system routes around. None of this is technically hard. It is a product choice, and right now the product choice cuts the other way: fluent, confident, complete-sounding answers test better, ship faster, and keep users engaged.

The policy implication is smaller and harder. Regulators asking whether AI systems are "accurate" are asking the wrong question, or at least an incomplete one. The right question is whether a system, in the hands of the population that will actually use it, produces better decisions than that population would have made without it. On the Marcoccia et al. evidence, at least when the advice is fluent and wrong, the current answer is no.

That is the stake. Not that the machines are wrong, they are sometimes wrong, as humans are, but that using them appears to make the humans stop noticing. A society in which a large share of everyday judgements are made by people who have quietly lost the ability to feel unsure is a different society. It is worth naming that before the deployments finish going in.

Footnotes