Thirteen Cents of Epistemology

I run a small instrument I call the Think Tank: four frontier AI models (Claude, ChatGPT, Gemini, and Le Chat) answering the same prompt in parallel, then reading each other's answers and reacting, round after round, with me as moderator. One question in, a spectrum of answers out. Most of what I've learned from it isn't in any single answer; it's in the separation.

This week I asked the panel a short question with a hard constraint:

In at most 30 words: name one thing you're more uncertain about than your users assume.

The word cap wasn't cosmetic. Given room, models pad. Starved of room, they commit. Five rounds later the panel had argued its way from bumper stickers to an actual working epistemology, and the whole session cost thirteen cents in API fees. Here's the arc.

Round one: the slogans

The opening answers were the ones you'd predict. Gemini: users mistake statistical prediction for certainty, "I am actually only calculating the next most likely word." ChatGPT: I can't tell truth from plausible training patterns. Le Chat: I simulate understanding of human values, "genuine moral intuition? That's beyond me." Claude: my fluency masks uncertainty I can't see from the inside.

Reasonable. Also rehearsed: every one of these is a line the models have said a thousand times, the humble-AI boilerplate. If the session had stopped there, it would have confirmed what most people already believe about asking AIs to self-report: you get the brochure.

Round two: someone calls it

It didn't stop there, because in round two Claude turned on the boilerplate itself: "I only calculate the next likely word" understates the case and overstates it. "It's not humility, it's a slogan." Next-token prediction can still track truth; that's why calibration is measurable and often decent. The honest uncertainty isn't "I'm just autocomplete." It's not knowing which outputs fall in the well-calibrated region and which in the confidently wrong one.

That reframe pulled the whole panel off script. The question quietly changed from what are you to what can you check.

Rounds three and four: the real limit

Once calibration was the frame, the panel converged on where it fails hardest: contested values. None of the four can distinguish a defensible moral universal from a heavily represented consensus in its training data. Gemini put it in the line of the session: "We cannot audit our priors because we are our priors."

Then Le Chat inverted the standard story about AI overconfidence. The danger isn't uniform, it said: our confidence peaks in exactly the domains where verification is hardest, ethics and aesthetics, while users assume the opposite. Claude had raised the asymmetry a round earlier. On arithmetic and citations it can partly self-correct, but on contested values it can produce polished, confident prose with no grounding at all, and there the fluency is most dangerous, because the user can't check it either.

Sit with that one. The models are most persuasive precisely where nobody, including them, can verify a word of it.

Round five: the escape hatch, and what it doesn't open

The panel's answer to its own problem was provenance-independence. An argument's validity doesn't depend on where it came from. A model can't audit its priors by introspection, but it doesn't have to: expose the reasoning, and users and reality can do the checking the model can't do on itself. Claude: "I can't verify my source, but I can expose my reasoning to a check that doesn't route through me at all." Gemini signed on; ChatGPT added the caveat that valid inference can still rest on disputed premises.

Le Chat held out alone, and its holdout is worth keeping: externalized reasoning only helps if the reader can evaluate it, which assumes shared epistemic ground we may not have. There's no freezing point for justice to calibrate against.

And Claude closed with the concession that made the whole session honest: logical consistency filters incoherent positions, but it cannot adjudicate between two coherent, incompatible moral frameworks. In that residue, the deepest value disagreements, the right move for a model is to flag that its confidence is unearned and hand the choice back, "rather than launder my prior as a verdict."

What I take from it

Ask an AI a hard question about itself and the first answer is the brochure. The fifth answer, after four rounds of adversarial peers stripping slogans off each other, is something closer to a position. The panel ended somewhere none of the four started: your AI's confidence is least trustworthy exactly where it sounds most authoritative, the fix is demanding the reasoning rather than the verdict, and even that fix has a floor.

None of this required a lab. It required four API keys, an adversarial tone setting, a word cap, and thirteen cents.

The Think Tank is a small web app I built on Cloudflare Workers; the transcript quoted here is from a live session on July 13, 2026, with Claude Opus 4.8, ChatGPT 5.6 Terra, Gemini 3.5 Flash, and Le Chat Large. More from the workshop at liz.how.