Shreshth Rajan, December 2025.
In June 1902 Gottlob Frege received a letter from Bertrand Russell. The second volume of his life's work was already with the printer. In about two pages Russell showed that Frege's system let you derive a contradiction, which means it could prove anything and so was worthless. Frege had worked inside that system for more than twenty years without noticing. He added an appendix to the volume saying that nothing worse can happen to a writer than to have one of the foundations of his work collapse once the work is finished.
Russell needed one letter. The flaw was invisible from inside the system and obvious from just outside it.
This is usually where Gödel gets invoked, and usually it's the wrong theorem of his. The one that fits is the second: a system strong enough to do arithmetic can't prove its own consistency. The one everyone quotes is the first, that such a system contains true statements it can't prove, and it gets stretched into a much larger claim.
The largest version is Penrose's, in Shadows of the Mind. Every machine is a formal system, so every machine has a true statement it can't prove. A human mathematician can see the statement is true. Therefore mathematicians aren't machines, and human-level AI is impossible. Lucas gave the argument in 1961 and Penrose sharpened it in 1994.
I think it's wrong, though the published objections mostly target the wrong step. Feferman argues that Penrose needs mathematicians to know something far stronger about their own reasoning than the theorem provides, and that he slides between the weaker and stronger versions without saying so. Chalmers makes the plainer point: the argument assumes we never err about what we can see to be true. Frege is the counterexample.
Both objections hold. But for AI the argument fails a step earlier, because its first premise is false. A frontier model isn't a formal system. It's ten to a hundred billion numbers tuned by gradient descent to predict text, with no axioms and no derivation rules anywhere in it. Gödel's construction needs a system that can make statements about its own proofs, and a model has no proofs. You can't even write the Gödel sentence down, because there's nothing to write it in terms of.
So Gödel doesn't put a bound on AI. The argument isn't defeated; it just never applies.
What does carry over is Frege's situation, and in a model you can actually measure it — as an analogy, not a theorem.
Every frontier model has a poor model of itself. Turpin and colleagues showed that a model's stated reasoning often doesn't match the computation that produced the answer, so it reaches a result one way and explains it another. Hubinger and colleagues built models carrying backdoors that survived safety training and that the model never mentioned. Arditi and colleagues found refusal running along a single direction in the model's internal state, which the forward pass can't observe while it runs.
None of these is the kind of bug a larger model removes. The description has to fit inside the thing being described, and that's the part that doesn't scale.
This is why I don't think alignment is a harder version of capability. Capability improves with scale, but self-knowledge doesn't. A bigger model is both better at its task and a bigger thing to model, so the part of it that would have to check the rest is always chasing a larger target. Making the model better makes that target bigger too.
So "the model will eventually verify its own alignment" has the structure backwards. A system doesn't check itself from the inside; the check comes from something that doesn't share its blind spot, whether that's a Russell, a held-out test, or a separate model. Penrose thought a mind could see past the limits of its own system. Expecting a model to check its own alignment is the same bet, and it fails because ultimately a better version of the system is still the system.