How ORA’s piece on Confidently Wrong, Together scored.

Accuracy

The headline figures (44% to 3%, 27% to 9%, near-doubling of confidence) match what the authors describe publicly, but the article miscasts the study as a single experiment with 700 participants when the authors describe five experiments with more than 3,200 (-15). The 27% to 9% accuracy drop occurred specifically when participants received systematically wrong AI advice, a condition the article does not name clearly (-8). The incentivised-condition 16% figure is not independently verifiable in public summaries and is presented without a source hedge (-5).

Balance

The piece has a clear editorial view but explicitly names the strongest objections: single preprint, not peer-reviewed, one model, one task, users may improve with familiarity. It grants those caveats before pressing on with its reading. Tone leans persuasive but the opposing case is stated in the authors' terms rather than strawmanned.

Concerns (3)

• majoraccuracy

  “It is a randomised experiment with 700 participants.”

  Materially misdescribes study design.

  Evidence: Authors publicly describe five experiments with more than 3,200 participants.

• majoraccuracy

  “accuracy fell from 27% to 9%”

  Omits that this condition used systematically wrong AI advice.

  Evidence: Author summaries specify the drop occurred under deliberately incorrect AI advice.

• minoraccuracy

  “accuracy in the incentivised AI condition reached 16%”

  Specific figure not visible in public summaries, no hedge.

  Evidence: Cannot corroborate this exact number from author or coverage summaries.

Reproducibility

Run

21 Jul 2026, 01:27 BST

Reviewer

claude-opus-4-7

Prompt SHA

206ee0cfb06c

Article SHA

ab69452c779e

Editor

ORA

Published

21 July 2026

Cost

$0.0000