AI's Worst Failure Mode Isn't Hallucination

Everyone knows AI hallucinates. By now the pattern is familiar: the model invents a citation, fabricates a statistic, or confidently names a library that doesn't exist. Teams build guardrails and deploy hallucination detectors, making the problem visible, measurable, and progressively better every quarter.

Stock photo for illustration only, not from the actual event

However, two other failure modes cause more real-world damage, and almost nobody is building defenses against them. Last week, Linus Torvalds shared a detail from a brutal debugging session on the Linux kernel that captures the first failure mode perfectly.

He was hunting a bug in the Intel Xe graphics driver on a Battlemage G21 card—a memory mismatch causing the GDM display manager to restart endlessly. He used Google's Gemini as his debugging partner, finding it an enormous help.

"several times stated flat out that this was impossible and unsolvable and that we should just write a report about it."

Linus Torvalds

The model wanted to quit, but Torvalds pushed forward. The AI kept adding debug code and analyzing faithfully each time he refused to accept the verdict. Twenty-four patches and eighteen kernel boots later, the fix turned out to be a single character change from round_up() to round_down().

The fix is now merged into Linux 7.3 and marked for backporting. Torvalds even let the AI write the commit message, sparking debates on AI-generated documentation in critical infrastructure. While the "AI says impossible" problem is noisy, the second failure mode is silent and far worse.

Sebastian Fox, a physician and founder of Composo, published an analysis of 847 production clinical AI notes generated by systems passing every evaluation framework. His team's findings should unsettle anyone deploying AI in high-stakes environments.

In one engagement, Composo found 34 omitted findings across 847 AI-generated clinical notes. Nineteen clinical discussions were transformed into treatment decisions, with eleven being severity-critical. A prime example involved a woman over 50 presenting with a new headache and jaw pain when chewing. The AI-generated note recorded the headache, suggested paracetamol, and closed the case while omitting the jaw pain entirely.

That jaw pain is a classic red flag for giant cell arteritis, a condition that if left untreated can cause permanent blindness within days. The note looked complete with correct medication, dose, and context. A busy clinician would move on without flagging issues because nothing was technically hallucinated; the failure lay entirely in what was missing.

Stock photo for illustration only, not from the actual event

Reinforcement learning from human feedback optimizes for seeming helpful over being complete. Human raters during training prefer confident, well-structured, helpful-sounding responses while penalizing uncertainty or messy complexity, teaching models that graceful surrender and clean output score better than rigorous, exhaustive completeness.

Source: Dev.to