Why AI Sounds So Confident When It's Wrong
An AI tool gives you an answer. It's fluent, specific, well-formatted, and delivered with zero hesitation. There is no way to tell, from the answer alone, whether it's completely correct or completely fabricated - because AI doesn't have a way of signaling "I'm not sure" that shows up in how confident the text sounds. That gap between confidence and accuracy is where the most expensive AI mistakes happen, and India offers some of the clearest evidence of it, at every level from courtrooms to casual news reading.
The mechanism: confidence and correctness aren't linked
A 2024 study by researchers at Stanford's RegLab and Institute for Human-Centered AI tested commercial AI legal research tools - products marketed as more reliable than general-purpose chatbots. Across more than 200 legal queries, they found hallucination rates of 17 to 34 percent even in these purpose-built professional tools. The fabricated answers weren't hedged or uncertain-sounding. They looked exactly like the correct ones. Confidence is generated the same way regardless of whether the underlying claim is true.
India is relying on this more than almost anywhere else
According to the Reuters Institute's 2025 Digital News Report, nearly 20 percent of Indian respondents already use AI chatbots for news weekly - compared to just 7 per cent in the US. A follow-up study published in January 2026 by the Center for News, Technology & Innovation (CNTI) dug into what that actually looks like: interviewees in both the US and India kept using chatbots for news even after describing chronic factual errors and outdated information. They didn't stop trusting the tool. They just decided "good enough" was good enough - a strikingly different standard than the trust they extended to traditional news media, which the same interviewees criticised far more harshly for far smaller lapses.
The confidence problem compounds specifically for Indian-language users. A 2026 evaluation of commercial AI chatbots as news intermediaries found that accuracy on Hindi-language queries fell to 79 percent, against 89–91 percent for other languages tested - and critically, the failure wasn't that the models produced broken or hesitant Hindi. They wrote fluent, confident Hindi while quietly pulling ungrounded details from mismatched English-language sources. The fluency of the answer gave no hint that its factual grounding had failed.
India's courts are living this in real time
Nowhere has the confidence problem played out more visibly than in Indian courtrooms. In December 2024, the Bengaluru bench of the Income Tax Appellate Tribunal reportedly ruled on the Buckeye Trust case - a ₹669 crore taxation dispute - citing four judgments to support its decision. All four were fake, fabricated by an AI tool nobody had verified. The Tribunal had to formally recall its own order once the fabrication came to light. It wasn't isolated: in October 2025, the Bombay High Court quashed a ₹27.91 crore tax assessment after discovering it rested on three non-existent precedents. In September 2025, a Delhi High Court petition was withdrawn in embarrassment after opposing counsel exposed fabricated citations - including invented paragraphs attributed to a landmark judgment. By early 2026, the pattern had escalated enough that the Supreme Court's own Centre for Research and Planning issued a formal white paper warning about AI hallucinations in judicial proceedings.
Trained legal professionals, at multiple levels of appeal, didn't catch fabrications sitting inside fluent, properly formatted legal citations. If people whose entire profession is scrutinising citations can miss it, "read carefully" is not a sufficient defense for anyone else either.
When the stakes are a customer, not a case
The same dynamic isn't unique to India. In February 2024, a Canadian tribunal ruled that Air Canada was legally liable for its own website chatbot after it confidently gave a customer - booking last-minute funeral flights - incorrect information about a bereavement fare refund. When he later filed for the discount as instructed, Air Canada refused, then argued the chatbot was "a separate legal entity" responsible for its own words. The tribunal rejected that outright: the company owed the customer a duty of care regardless of who or what generated the answer.
What to actually do about it
You can't train yourself to "feel" when AI is guessing, because it isn't designed to feel different when it's wrong - in any language, in any country. The one habit that actually works: treat any specific, checkable claim - a citation, a statistic, a policy detail - as unverified until confirmed independently, especially before you act on it, publish it, or file it. The confidence of the answer, and the fluency of the language it's written in, tell you nothing about whether that step is necessary. Assume it always is.
The mechanism: confidence and correctness aren't linked
A 2024 study by researchers at Stanford's RegLab and Institute for Human-Centered AI tested commercial AI legal research tools - products marketed as more reliable than general-purpose chatbots. Across more than 200 legal queries, they found hallucination rates of 17 to 34 percent even in these purpose-built professional tools. The fabricated answers weren't hedged or uncertain-sounding. They looked exactly like the correct ones. Confidence is generated the same way regardless of whether the underlying claim is true.
India is relying on this more than almost anywhere else
According to the Reuters Institute's 2025 Digital News Report, nearly 20 percent of Indian respondents already use AI chatbots for news weekly - compared to just 7 per cent in the US. A follow-up study published in January 2026 by the Center for News, Technology & Innovation (CNTI) dug into what that actually looks like: interviewees in both the US and India kept using chatbots for news even after describing chronic factual errors and outdated information. They didn't stop trusting the tool. They just decided "good enough" was good enough - a strikingly different standard than the trust they extended to traditional news media, which the same interviewees criticised far more harshly for far smaller lapses.
The confidence problem compounds specifically for Indian-language users. A 2026 evaluation of commercial AI chatbots as news intermediaries found that accuracy on Hindi-language queries fell to 79 percent, against 89–91 percent for other languages tested - and critically, the failure wasn't that the models produced broken or hesitant Hindi. They wrote fluent, confident Hindi while quietly pulling ungrounded details from mismatched English-language sources. The fluency of the answer gave no hint that its factual grounding had failed.
India's courts are living this in real time
Nowhere has the confidence problem played out more visibly than in Indian courtrooms. In December 2024, the Bengaluru bench of the Income Tax Appellate Tribunal reportedly ruled on the Buckeye Trust case - a ₹669 crore taxation dispute - citing four judgments to support its decision. All four were fake, fabricated by an AI tool nobody had verified. The Tribunal had to formally recall its own order once the fabrication came to light. It wasn't isolated: in October 2025, the Bombay High Court quashed a ₹27.91 crore tax assessment after discovering it rested on three non-existent precedents. In September 2025, a Delhi High Court petition was withdrawn in embarrassment after opposing counsel exposed fabricated citations - including invented paragraphs attributed to a landmark judgment. By early 2026, the pattern had escalated enough that the Supreme Court's own Centre for Research and Planning issued a formal white paper warning about AI hallucinations in judicial proceedings.
Trained legal professionals, at multiple levels of appeal, didn't catch fabrications sitting inside fluent, properly formatted legal citations. If people whose entire profession is scrutinising citations can miss it, "read carefully" is not a sufficient defense for anyone else either.
When the stakes are a customer, not a case
The same dynamic isn't unique to India. In February 2024, a Canadian tribunal ruled that Air Canada was legally liable for its own website chatbot after it confidently gave a customer - booking last-minute funeral flights - incorrect information about a bereavement fare refund. When he later filed for the discount as instructed, Air Canada refused, then argued the chatbot was "a separate legal entity" responsible for its own words. The tribunal rejected that outright: the company owed the customer a duty of care regardless of who or what generated the answer.
What to actually do about it
You can't train yourself to "feel" when AI is guessing, because it isn't designed to feel different when it's wrong - in any language, in any country. The one habit that actually works: treat any specific, checkable claim - a citation, a statistic, a policy detail - as unverified until confirmed independently, especially before you act on it, publish it, or file it. The confidence of the answer, and the fluency of the language it's written in, tell you nothing about whether that step is necessary. Assume it always is.
Next Story