The Machines Now Decide Who To Believe. Nobody Audited The Evidence.
There’s a marketing report making the rounds right now with a tidy little thesis: in a zero-click world, the goal is no longer to be found, it’s to be believed. It’s a clean line. It fits on a slide. It will be quoted in roughly four thousand LinkedIn posts before this sentence finishes loading.
And it’s correct, in the way a weather forecast is correct. It tells you it’s raining. It does not tell you the dam upstream is cracking.
Here’s the thing the believability framing quietly skips over: believed by whom, exactly?
Because the new arbiter of credibility isn’t a person who read your article and nodded. It’s a model. A model that doesn’t read so much as retrieve, doesn’t reason about truth so much as pattern-match against whatever it was fed, and doesn’t know the difference between a peer-reviewed study and a confident blog post wearing the same outfit. We have, more or less by accident, handed the job of “deciding what’s trustworthy” to a system whose own evidence base is quietly turning into a hall of mirrors.
That’s the actual story. Not “be more believable.” It’s: credibility is becoming a closed loop, machines are policing it, and the data underneath is starting to rot.
I’ve spent about two decades building systems that were supposed to be sources of truth and watching them, without fail, slowly become sources of confident nonsense at scale. So let me walk you through what’s really happening here. It’s more interesting than the slide.
First, the loop nobody wants to look at
Start with a fact that should bother you more than it does.
In July 2024, a team out of Oxford, Cambridge and Toronto published a paper in Nature with a wonderfully blunt title: “AI models collapse when trained on recursively generated data.” The finding, stripped of the math: when you train a generative model on the output of generative models, it degrades. Not gracefully. The rare stuff, the edges, the weird-but-true tail of human knowledge, vanishes first. The model converges toward a bland, confident average and then toward gibberish. The researchers called it model collapse, and showed it happens across language models, image models, the lot. Indiscriminately train AI on AI, and the tails of reality quietly disappear.
Now hold that next to a second fact: the open web is filling up with AI-generated text at a rate nobody can fully measure but everybody can feel. The blog posts. The product descriptions. The “ultimate guides.” The SEO sludge that reads like it was written by an intern who has never experienced an emotion.
Put those two facts in the same room and you get the loop below. AI writes the web. The web becomes the training and retrieval data. The next model learns from it. Repeat. Each lap, the signal loses a little blood.
It’s a snake eating its own tail, except the snake is also writing thought-leadership about how nutritious the tail is.
The uncomfortable part isn’t that this might happen. It’s that the economic incentives guarantee it. Generating content is now nearly free. Verifying it is not. When one side of a market collapses to zero cost and the other stays expensive, you already know which one wins by volume.
The “believability” report has the right symptom and the wrong patient
Let’s be fair to the marketing crowd. They’ve noticed something real. Showing up in an AI answer is now table stakes, and being trusted inside that answer is the new contested ground. That’s true.
But their prescription, build a “consistent narrative,” accumulate “third-party validation,” create a “chorus of content” across platforms, is essentially a recipe for manufacturing the appearance of credibility at scale. Coordinated narrative. Corroboration across channels. Repetition until the machine reads it as durable truth.
Read that again and notice what it is. It’s not a plan to be more truthful. It’s a plan to be more legibly truthful to a model that confuses repetition with reliability. It’s reputation laundering with better production values.
I’m not saying anyone’s being cynical. I’m saying the framing treats the model as a neutral judge you simply need to persuade, when the model is actually a deeply gameable system that rewards exactly the behavior that pollutes it. You optimize for “the machine finds this credible,” and you get an internet engineered to look credible to machines. Which is, if you squint, just SEO with a philosophy degree.
The honest question isn’t “how do I become believable?” It’s “what is this thing actually using to decide, and can it tell the difference between earned trust and a well-funded echo?”
Spoiler. Mostly, it can’t.
How the machine actually picks who to trust
Here’s where my inner architect gets twitchy, because the mechanism is both simpler and dumber than people assume.
When an AI answers a question with sources, it’s usually running some flavor of retrieval-augmented generation. There’s a pipeline. And every stage of that pipeline has a bias baked in.
The retrieval system grabs candidate documents. A re-ranker scores them. The model picks a handful to “ground” its answer. Then it writes something fluent and slaps citations on it. Four stages, four sets of fingerprints on the glass.
What survives that gauntlet? Research from late 2025 found something that should give every publisher and every PR team pause: across six major AI search systems, fewer than ten distinct URLs showed up in eighty percent of all answers. Eighty percent. The “infinite open web” collapses, at the moment of retrieval, into a tiny cartel of usual suspects. The long tail of human knowledge isn’t being weighed and found wanting. It’s not being weighed at all.
And it gets better, by which I mean worse. LLMs exhibit a documented citation bias toward already-highly-cited work, even controlling for year, venue, and author count. It’s the Matthew effect with a server farm: to those who have citations, more shall be given. The famous get famouser (if that is even a word). The obscure-but-correct stay obscure. If you were hoping the machines would be a great democratizing force for overlooked truth, I have a recursively generated bridge to sell you.
Here’s the gauntlet. Once you see the shape of it, you can’t unsee it.
Notice what that funnel really is. It’s not a search for truth. It’s a search for whatever was easy to retrieve, recent, well-structured, and already popular. Four reasonable-sounding engineering choices that add up to a machine with strong opinions about whose voice counts, and no opinion at all about whether they’re right.
The part that should keep you up at night
So the sources get narrow. Fine. At least the model is faithful to the ones it picks, right?
Naah.
A 2025 study in Nature Communications looked at seven major LLMs across medical queries, the kind of questions where being wrong has consequences measured in human beings. The finding: somewhere between fifty and ninety percent of LLM answers were not fully supported by the sources they cited. The citations were there. They looked great. They just didn’t actually back up the claim sitting next to them.
Sit with that. The model retrieves a real source, displays a real link, and then writes a confident sentence that the source does not support, with a citation that looks like proof. It’s the homework-citation move we all pulled in school, where you bolt a reference onto a sentence you made up and hope the teacher doesn’t check. Except now it’s automated, fluent, and trusted by millions of people who absolutely do not click the link. LIKE WHAT?
Separate research on deep-research agents found citation hallucination rates running anywhere from eleven to fifty-seven percent across deployed commercial systems. The citation is the trust signal. The trust signal is frequently fictional.
We built machines that produce the visual grammar of credibility, the links, the confident tone, the tidy summary, while quietly decoupling it from whether any of it is true.
This is the real credibility paradox, and it’s a lot darker than “be believable.” The very features that make AI answers feel trustworthy, the citations, the synthesis, the calm authoritative voice, are the features most easily faked, both by the model and by anyone optimizing to be cited by it.
Meanwhile, the people who made the evidence are being quietly defunded
Here’s the loop closing in a way that’s almost too on-the-nose to be real.
The evidence AI relies on, real reporting, real research, real first-hand expertise, costs money to produce. That money came, for two decades, from people clicking through to the websites that made it. That social contract was simple: publish, get visited, get paid, publish more.
Zero-click is dissolving that contract in real time. By late 2025, somewhere around sixty percent of Google searches ended without a single click to an external site. When an AI Overview shows up, position-one click-through rates dropped by roughly fifty-eight percent year over year, by some measures. News sites lost a quarter of their search traffic in a year. Business Insider’s organic search traffic fell more than half. Educational publishers watched homework-help traffic evaporate into the Overview that ate their content to answer the question.
Follow the logic to its end. The machine summarizes the source. The summary removes the reason to visit the source. The lost visit removes the revenue. The lost revenue removes the source. And the next generation of models, hungry for fresh, real, human-grounded evidence, finds a little less of it each year, and a little more of its own recycled output waiting where the real thing used to be.
That’s not a content strategy problem. That’s an ecosystem metabolism problem. We’re optimizing the harvest while quietly salting the field.
So what does an architect actually do about it?
I’m allergic to articles that diagnose a civilizational problem and then end with “and that’s why you should post more on LinkedIn.” So let me be useful, from the builder’s chair rather than the marketer’s.
If you’re making content or running a brand, the believability crowd isn’t wrong that you need to be machine-legible. Structure your evidence so it survives chunking. Put your real data and real claims in clean, self-contained, quotable units. Be the primary source, not the eleventh summary of one. That’s table stakes, and it’s fine advice as far as it goes.
But the deeper move, the one almost nobody is making, is this: stop treating the model’s judgment as ground truth and start treating it as a measurement instrument with known biases. You wouldn’t trust a thermometer without knowing its calibration. The model favors recency, popularity, structure, and confident phrasing. Those are its systematic errors. Once you know a system’s biases, you can design around them instead of worshipping the output.
If you’re building with these systems, the lesson is sharper. Provenance is no longer a nice-to-have feature you add in v2. The question “where did this claim actually come from, and does the source actually say it?” has to be a first-class part of the architecture, not a citation cosmetically appended at the end. Verify that cited sources are reachable, relevant, and actually consistent with the claim. Treat synthetic data in your pipeline like you’d treat a known contaminant: useful in measured doses, catastrophic when you stop filtering. The Nature collapse paper’s own authors said as much, training on AI output isn’t impossible, but the filtering has to be taken deadly seriously.
And if you’re just a human trying to know things in this environment? Develop a slightly rude reflex. When a machine hands you a confident, beautifully-cited answer, the citation is the least trustworthy part, not the most. Click the link. Half the time, the link doesn’t say what the sentence says. That single habit puts you ahead of most of the internet.
The strange new shape of trust
Step back far enough and you can see the thing changing underneath all of this.
For most of human history, credibility was social. You trusted a person, an institution, a track record, a face. Then it became algorithmic, you trusted whatever ranked first. Now it’s becoming synthetic: you trust a fluent summary assembled by a system that has no concept of trust, optimized by people who’ve learned exactly which buttons make it cite them, fed by a web increasingly written by machines like itself.
Each layer abstracted us a little further from the original act of one human verifying something and vouching for it. We didn’t remove the need for that act. We just hid it behind enough interface that everyone assumes someone else is doing it.
Nobody is doing it.
The believability report is right that the goal is no longer to be found. But it stopped one question short. The goal is to be believed, yes, and the quiet horror underneath is that the thing doing the believing was never built to know what’s true. It was built to sound like it does. We’re all now optimizing, frantically, to be trusted by a mirror.
And perhaps the strangest part is this: the better these systems get at sounding credible, the less anyone bothers to check whether the evidence underneath ever existed. Eventually the loop closes completely, machines writing the web, machines reading the web, machines vouching for machines, and somewhere in the middle a confident sentence with a broken citation that everyone believes because it was too well-formatted to question.
That’s the moment the data stops being a record of what humans found true.
That’s when software stops behaving like a library, and starts behaving like a rumor with infrastructure.
-Hardik
If this rewired something in how you read AI answers, that was the point. I write about the systems underneath the hype, the cognition, the memory, the quiet failure modes nobody puts on the slide. Subscribe if you want the next one.
buymeacoffee.com/HardikGoel