Answer engine summary
Automation bias is the tendency to accept a system's output rather than scrutinise it — and it is the main reason human-in-the-loop safeguards weaken over time.
The definition
Automation bias describes two related tendencies: accepting an automated recommendation without adequate checking, and failing to act when the system does not prompt you to.
Both are ordinary human responses to reliable tools, not negligence. That is what makes them difficult to design around.
Why volume makes it worse
Vigilance is expensive and does not scale. A reviewer who has seen four hundred correct outputs is not being careless when they approve the four hundred and first quickly; they are responding rationally to their experience.
What actually helps
Measures that work tend to be structural rather than motivational: sampling audits, deliberately surfaced uncertainty, requiring an independent judgement before the system's output is shown, and limiting caseload.
Training people to be more careful, on its own, reliably does not.
Why fiction cares
Because it explains how sensible people end up ratifying harm without anyone behaving badly, which is the exact dramatic problem that modern AI fiction has to solve.
“Useful was not the same as right.”
Questions
Is automation bias proven?
It is a well-established finding in human factors research, studied across aviation, medicine and other high-stakes automated settings.
Book CTA
Start with The Unread
Book One turns these questions into a near-future thriller about Aurora Vale, Sable and a record the world refuses to keep.