What nobody can tell you
Stop on this before anything else, because it is the strangest fact in the whole field and it has been worn so smooth by repetition that people walk straight past it without seeing it. The people who built these machines do not know why they work.
Be careful what that claim is, because it is easy to soften into something ordinary, and softened it loses all its force. It does not mean the builders cannot tell you how the machine is wired; they can draw the architecture for you down to the last connection. It does not mean the operation is mysterious; the operation is almost insultingly simple, and you will have the whole of it in a few minutes. And it does not mean that anyone has overlooked the fact that the thing was trained on a mountain of human writing; that is the first thing everyone says.
"It read the whole internet and got smart" is not an answer. "It is just statistics, it predicts the next word" is not an answer either, because that names what it does and says nothing about why doing that should produce a single line worth reading. "It has hundreds of billions of parameters, like the neurons in a brain" restates the machine and explains nothing at all. "The abilities emerge at scale" gives the surprise a label and leaves it exactly as surprising as it was.
Past every one of those, the real question stands untouched: not how the thing is built, not what it does, but why doing that should work at all. Why should fitting a machine to the statistical shape of human text produce essays and proofs and working code and a line that lands in a room, when fifty years of building language the obvious way, as a system of rules, produced almost nothing?
This is not a question only outsiders ask, and the builders say it themselves, plainly and on the record. The flagship technology review of one of the world's great engineering schools reports that for all its runaway success, nobody knows exactly how, or why, it works. A computer scientist at Harvard, seconded to one of the leading labs, compares the moment to physics at the start of the twentieth century: a heap of experimental results that surprise the experimenters and resist every attempt to explain them. The company that built one of the largest of these systems states, in its own research, that it does not understand how its models do most of what they do. A young researcher arriving in the field asks her teachers why the systems work and is told there are no good answers.
The admission is genuine, it comes from the people standing closest to the machine, and the field has filed it as an engineering puzzle that more tools and more scale will eventually close.
It is not that kind of puzzle, and this piece is the long case that it is not. The reason the builders cannot say why their machine works is that the answer was never going to be found in the machine. The answer is in the thing the machine was fitted to. The question only looks like a question about computers; it is a question about language, and about the thing you do every time you open your mouth.
The line that moved the room
Type "See you at the" into your phone and it offers you "office," "party," "station." You have used this so many times that you no longer notice it, and most days you let it finish the small words for you, because most days it is right.
That little guesser, the one you half-trust to end your sentences, is the whole of it. A language model is that, and nothing else, carried to an extreme: where the phone offers one word, the larger machine continues for paragraphs, and the operation underneath is the one the phone runs, predicting what should come next in a run of text.
Now hold that next to something else you have watched it do. You asked it, late, for a few lines to read at a leaving do, or a note for someone who had just lost a parent, and it gave you something you actually used, something that caught a little in the throat of the room. Or it wrote a stretch of code that ran first time, or explained a hard idea to a child in the same breath it drafted a contract. People who do those things for a living looked at the output and could not tell it from their own.
And the machine that wrote the line that moved the room did not know what a single word of it meant. It did not know the person had died, and it did not know what dying is. It ran the same next-word guess your phone runs, over and over, and out came something that landed.
That is the unease, and it is the right unease, and almost everything written about these systems spends its energy talking you out of it. This piece does the opposite: it takes the unease and follows it all the way down. But to follow it you first have to see, plainly and without the words that flatter it, what the machine is actually doing.
What the machine actually does
Watch it work on a single phrase, because the whole engine is visible in three words. Give it "The cat sat." It reads those words, runs them through the patterns it absorbed in training, and assigns a probability to every word it knows as a candidate for what comes next. Say "on" comes out highest, with "under," "near," "beside" trailing. It picks one, the text is now "The cat sat on," and it runs the entire procedure again over the longer phrase, and again, and again.
A sentence, a paragraph, an essay, a proof, an apology: all of it is this one move repeated, predicting the next word, adding it, predicting again.
What does the heavy lifting is how those probabilities get computed, and the history of that is short and tells you something. The first language programs of the 1950s simply counted: if "good" came before "morning" often enough, the machine learned to expect "morning." Crude, but it set the ground, which was that language could be handled by counting rather than by understanding.
For fifty years after that the best minds in computer science tried the other road. They tried to build language as a logic, a system of rules and stored facts, a machine that represented what a sentence meant inside itself and reasoned over it. They were not fools and they were not underfunded, and the attempt came to nothing, systematically, for half a century. The programs that parsed sentences into logic collapsed the moment language stepped outside their toy world. The project to write common sense down as millions of hand-coded facts ran for decades, spent fortunes, and produced a system that could not hold a conversation.
Then, in a single decade, the machines that gave up on rules and simply fitted themselves to the statistical shape of human writing did what the rule-writers never could. The turn came in 2017, with an architecture that stopped reading a sentence word by word in order and instead looked at all the words at once, letting each one weigh its relation to every other. After that the only thing that really changed was size. One model in 2020 ran a hundred and seventy-five billion internal numbers over three hundred billion words of text and produced prose that professionals could not reliably tell from human writing. By 2022 a chat window put it in front of everyone, and a hundred million people were using it inside two months. The same system passes the bar exam near the top of the range, writes working code, argues philosophy, and produces poetry that moves competent readers, with no switch thrown between the tasks.
The history holds the clue, and the clue is in the shape of the collapse. For fifty years, building language as a logic produced nothing. Then giving up on logic and fitting a shape produced everything, with almost nothing changed but the scale. If language were the kind of thing a mind grasps by knowing its rules, the rule-writers would have won, because encoding rules was exactly what they were doing. They lost, and the loss tells you what language is. And if language is not a logic, then fitting its shape is not understanding it, any more than a plaster cast of a footprint understands the foot. The machine generates what it generates by not doing the thing we keep saying it does.
So be exact about what "it does not understand" denies, because there are two ways to understand a thing and the machine has neither. There is the way you understand, with a lived sense of what the words are about. And there is the cooler way a logician understands, working from definitions, applying a rule to a case, deriving a conclusion step by step. The machine does neither; it holds no definitions, checks nothing against a stored picture of the world, runs no proof. When it gives the right answer to a sum, it has not worked the sum the way even a pocket calculator works it. It has continued the text with whatever the patterns made likeliest, and where the right answer was the likely continuation, the right answer is what appeared.
The claim is not that it thinks in some thin, dry, machine way instead of a warm human one. Neither kind of thinking is happening anywhere in it.
Hold on to that, because the real question is hiding inside it. How can guessing the next word from learned patterns, with no understanding of either kind anywhere in the process, produce work we cannot tell from the work of understanding? The answer will not be about the guesser; it will be about the thing it is guessing at.
A genuine achievement, said plainly
Before the dismantling, the achievement deserves its due, because the rest of this only works if you trust that it is not a sneer.
What the engineers did is real engineering of a kind the field has rarely seen at this scale. They cracked problems that had held for half a century. The capabilities in front of us now would have read as science fiction ten years ago, and the people who built the last generation are routinely surprised by the next.
What they built is best described by an old pattern: it is aviation before aerodynamics, medicine before biochemistry. A working artefact, built and shipped and used by millions, without the underlying science that would explain why it works. The author of this piece uses one of these systems most days and would not willingly give it up. Nothing here is an argument that the thing is useless, or fake, or that you should stop using it. The argument is narrower and stranger than that, and it is about what to call the thing they made, and where the part that looks like understanding actually lives.
The builders, asked what they made, reach for the words their training hands them. They come from mathematics and formal systems and engineering, and a mind never meets a new object empty-handed; it sees through the frames it already carries. Through the frame of formal systems, a machine that produces fluent answers looks like an engineered intelligence, a mind assembled out of maths, and so that is what it gets called: we built an intelligence.
That is not vanity, and from inside the discipline that produced it, it is the natural reading. And yet the very people who built it, asked how it actually works, say they do not know. They know how it is constructed and cannot say why what comes out reads as understanding. That gap is the door this piece walks through, and it does not dispute the achievement; it asks what the achievement consists in.
There is a test that settles the question, and the machine takes it every day in plain sight. If the system understood, its mistakes would be a mind's mistakes: it would falter where the material is hard in earnest, grow careful where the stakes are high, and above all it would know the difference between saying something true and saying something that has the sound of truth without the substance.
It does none of this, and the reason is the telling one. It lays out a flawless derivation and then states a false conclusion with the identical confidence, drawn from the identical machinery, because there is nothing behind either one to hold it to account. A thing that produces true sentences and convincing falsehoods by the very same process has no way, from the inside, to tell them apart, and you can watch it take that test and come up short all day. That is not the signature of a mind that occasionally errs; it is the signature of a system that was never dealing in truth at all, only in the shape that truth, among other things, leaves in writing.
So the real question is not about the machine at all. It is about us, and about the thing we make every time we speak. What is human language, such that the residue it leaves in plain text is rich enough for a machine that understands nothing to wear it and pass?
The mind comes in with the language
To get there, one assumption has to be set down, because it is the assumption that makes these systems look impossible. The assumption is that meaning is information kept inside the head, that somewhere in a person sits the meaning, finished, and that words are the wires that carry it out. If that were the whole truth, a machine with no inside, no sensations, nothing private at all, could not mean anything by the words it produces, and the unease you started with would be the end of the story.
Two ordinary facts sit badly with that whole picture.
The first comes from the children who never received language in time. In the storybook the child raised by wolves grows up noble and fluent, and in life it does not happen. A boy was found in the woods of southern France around 1800, perhaps eleven years old, and after years of patient teaching he could read a single word, "milk," and no more. In 1970 a girl later called Genie was found in Los Angeles, where she had been kept in near-total isolation from the age of about twenty months until she was thirteen. After her rescue she picked up hundreds of words quickly, and grammar never came; she stayed at two-word strings and could not tell "the boy pushes the girl" from "the girl pushes the boy" better than chance.
These cases sit badly with the idea of a complete self waiting inside for words to be bolted on. They suggest the opposite, that much of what we mean by a human mind is not there in advance but comes in with language, and that past a certain age it does not come at all.
The second fact is so ordinary it is easy to miss. Almost every child assembles the entire working machinery of a language before the age of four, the tenses, the clauses nested inside clauses, the agreements between parts, with no rule ever taught and no correction that could do the work. Nobody drills a three-year-old in grammar, and it settles on its own, out of nothing but being surrounded by people using words. What the child takes in is not a set of rules handed over as information; it is a practice, absorbed by living inside it.
Both facts lean the same way, and they lean hard. They make it hard to picture a person as born holding meanings inside, with language the cable that carries them out, and easy to picture meaning instead as something a person is drawn into, from a public world of language that was running long before they arrived.
This matters for the machine, and it matters directly. To whatever extent meaning was never simply stored content inside even the humans who have it, the absence of such a store inside a model is less obviously the disqualification it first seemed. The place meaning has always lived is the public practice and the record it leaves in speech and writing, and that record is exactly what the model is trained on.
It reaches even into the most private corner you have, the silent reading you are doing right now. For most of the history of writing nobody read this way. Reading meant reading aloud, the eyes feeding the voice, the sense carried on the breath in a room where others could hear. When a bishop around the year 400 read with his eyes moving and his voice still, a man watching stopped to write it down as something strange enough to need explaining.
The texts themselves ran the letters together with no spaces between the words, so a reader had to sound them out to find the joints. The gaps you are reading through right now, the spaces that let a silent eye cut the line into words without a voice, were not there; they were added later, by scribes, centuries on. It was that change in the marks on the page, as much as anything in the skull, that made fast silent reading ordinary.
Even the most inward act language has turns out to be the public world folded inward, shaped by a change in the shared technology of writing. Once again, the inside leaning on the outside for its shape.
The words that smuggle a mind into the machine
With that in hand, look at the machine again and watch the ordinary vocabulary do its quiet work. We say the model reasons, knows, understands, plans, decides. The words are not lies and they are not quite metaphors; they are borrowed from the description of human conduct and laid over a process that has none of the structure they normally report, and once you look at the process the borrowing shows.
Start with the plainest fact about a trained model: it is frozen. When training ends, the model is a fixed array of numbers, and they do not change from one use to the next. The same input under the same settings gives the same output every time. The numbers on Tuesday are the numbers from Monday. Nothing inside accumulates or remembers between one response and the next. Whatever it seems to do when it appears to think, it does with a frozen object that holds no inner state across a conversation beyond the text of the conversation itself, which is fed back in, in full, as the next input.
There is no inner room where it keeps its thoughts; the text on the screen is the only memory there is.
This bears directly on the word "reasoning." When a model is said to reason through a problem, what happens is that it emits a run of words that has the form of a chain of reasoning, because chains of reasoning are everywhere in the training text and it has learned their shape. When you tell it to think step by step and it produces a visible chain of steps, the steps are not a window onto some prior inner working it performed and is now reporting. The steps are the working, in the only sense in which it computes anything.
They are words generated one at a time, each conditioned on the ones before, and the reason this improves the final answer is mechanical: the intermediate words become part of the input, lengthening the run and steering what comes next towards the regularities that, in the training text, tended to follow correct working. The model is not showing you its reasoning; the text it generates is doing the work reasoning does, by occupying the page and conditioning what follows, with no reasoner behind it.
That is exactly why it can produce a flawless chain of steps and then a wrong answer, or a wrong chain and a right answer. The steps and the answer are each produced by the same guessing process, and there is no inner commitment binding them that could make them agree.
The same goes for what we call knowing when the model answers. When it answers a factual question correctly it has not consulted a store of facts it believes. It has continued the run with the word the training made likeliest, and where the training text reliably paired that question-shape with that answer-shape, the likely word is the true one. Where it did not, or where the likely continuation has the shape of an answer rather than the substance, it produces a confident falsehood with exactly the same machinery and exactly the same inner signature of correctness, which is to say no signature at all.
It has no way, from the inside, to tell a continuation that is probable because it is true from one that is probable because it sounds like an answer. The two are not marked apart anywhere in the process, because the process tracks one thing, the likelihood of the next word, and nothing else.
None of this means the thing is simple or that the shape of language is shallow. The shape is staggeringly intricate, and why it is intricate enough to carry what it carries is the whole of the rest of this piece. The claim is only descriptive, and the investigation rests on it: the system is a fixed function that, run over and over, predicts the next word from the distribution of the text it learned, and the words reasoning, knowing, understanding, thinking, when used of it, name the shape those activities leave in text rather than the activities themselves.
If a process this thin produces what these systems produce, the richness is not in the process; it is in the material the process runs on, which is to say in language.
So the question becomes: what is language, that its residue is this rich? To answer it we put language through the people who, over the last century, gave the most exact account of what it is. You will not need to have read any of them. Each idea is built from an ordinary example you can check against your own speech, and the name comes only once the idea is already in your hand.
The picture of language we all inherited, and why it is wrong
There is a picture of language we all carry without having put it into words. Meaning is a content that sits inside the head; words are vehicles that carry it from one head to another; talking works when the content arrives intact. Language, on this picture, is a delivery service and a sentence is a parcel with something packed inside.
It feels obviously true, and it is the picture the whole field was built on. If language really worked this way, a safety filter or a fact-checker would only have to open the parcel, read the contents, and decide.
Take the word "but," which names nothing and points at no object the way "table" points at a table. Yet put "the model is accurate and slow" next to "the model is accurate but slow." The facts stated are identical and the meaning is not. The "but" adds a turn, an expectation that accuracy and speed should have come together, so that the slowness lands as a price rather than as a second fact sitting calmly beside the first.
That turn is not packed inside the word at all. The word performs a relation, and the relation happens between the word and the situation it is used in. If a three-letter conjunction already breaks the delivery picture, the picture was never going to hold.
The reason was found more than a century ago by a Swiss linguist named Ferdinand de Saussure, the man who founded the modern study of language. A word does not mean by pointing at a thing; it means by sitting in a web of differences from all the other words. "Cat" is what it is because it is not "dog," not "rat," not "cap," not "bat." Pull one word out of the web and the value of every other word moves a little, because each word's meaning is a matter of how it differs from the rest. Meaning is not a label stuck on an object; it is a position in a system, and the system is what gives the position its value.
Anyone who has built a sentiment classifier has run into this in the hardest possible way. The model learns that "good" goes with positive and "bad" with negative, and it works, until a teenager types "that film was sick." Every signal in the word says illness, disgust, the negative pole, and the teenager means the opposite, means it was magnificent. The word carries no fixed nugget of meaning that the model can read off it. "Sick" takes its value from the whole system it is living in, from a generation's worth of inversion and play, and the same four letters point one way in a medical note and the other way in a group chat.
The expression's value belongs to the system, and not to anything stored inside the word.
And the system does not hold still, which anyone who has chased a word through a thesaurus has felt. You look up one word, follow a synonym, follow one of its synonyms, and a few steps later you are somewhere unexpected, among words that share something faint with where you began but have drifted somewhere else entirely.
The chase never lands, and there is no final word at the end that sits still and says, here, this is what it really meant. The meaning is always one step further on than you are.
You can feel the same thing inside a single sentence. "Meet me at the bank," she wrote, and I pictured marble floors and a queue, until the next line came: "by the willow, where we used to swim." The bank was never the one I had built in my head. The word sat there, unsettled, and the words that came after reached back and decided it.
And it is never finally safe even then, because the message could go on, "they have filled the river in now, it is all car park," and the willow drowns under tarmac and the picture turns again. Each new word reaches back and reworks the words already passed, and you never arrive at a meaning that sits still.
This was named by a French philosopher who spent his career on exactly this problem, Jacques Derrida: meaning is endlessly deferred, pushed forward, never wholly present in the moment a word appears, because every word leans on the ones around it, the ones before and the ones still to come, and on all the absent words it is quietly not.
If meaning were a label on a thing, a machine could read the label the way a customs officer reads a declaration. Meaning is a position in a shifting web, and the web reorganises when the surroundings reorganise, and the surroundings are not in the string of characters at all. They are in the living activity the string came from, and a machine that has only the string has only the trace of that activity, never the activity itself.
Which is the first hint of the answer this piece is after. The richness the model is trading on is the richness of a web that never stops moving, and the reason that web is rich enough to carry fluent prose is that its meaning can never be pinned, listed, finished. If meaning could be arrested in a final form, the patterns would be thin and a rule-book could hold them. Because it cannot, the patterns are dense beyond any rule-book, and dense enough to be worth approximating.
The beetle in the box
There is a deeper assumption under all of that, and it is the one the unease really rests on. It is the assumption that meaning is something that happens inside, in private, that when you understand "pain" or "red" or "grief," there is an inner object, a sensation, a mental content, and the word is the public label you hang on the private thing. If that is true, a machine with no inside cannot mean anything, and there is no one home to do the meaning.
The man who took this picture apart had built the most rigorous version of it himself, earlier in his life, and then spent his last years demolishing it from the inside. This is Ludwig Wittgenstein, the philosopher who gave the twentieth century its sharpest account of what language is, and his decisive move is a thought experiment that dissolves the unease at its root rather than arguing with it.
Imagine, he says, that everyone carries a small box, and inside each box is something the owner calls a "beetle." No one can ever look inside anyone else's box. You know what a "beetle" is only from your own.
Now the people use the word "beetle" among themselves, in ordinary talk. And here is the turn: the thing in the box could be different for every person, it could be changing from moment to moment, one box could be empty, and none of that makes the slightest difference to how the word "beetle" works in the shared conversation, because the word does its work in the talk between people, and the private contents, whatever they are, never enter that exchange. The object in the box, he says, drops out of consideration as irrelevant. Whatever the word means, it cannot be the private inner thing, because the private inner thing makes no difference to the use.
Sit with what that does to the question this piece has been circling. The meaning was never the object in the box; it was never inside the head at all. It lives in the public practice, in the shared, correctable activity of using words with other people.
And here is the hinge for the whole question. If meaning lived inside, the model would be locked out for want of an inner life, and the unease would be correct. Meaning lives outside, in the use, and the use leaves its full record in public text, which is the one thing the model has. The model is shut out of the box, and it turns out there was never anything in the box that mattered. What mattered was always on the outside, in use, where the writing is.
Use is what Wittgenstein puts in place of the inner picture. The word "water," shouted by a firefighter at a blaze, is a command. Whispered by a patient in a hospital bed, it is a plea. Called out by a geologist in a desert, it is a discovery. Intoned by a priest at a font, it is a blessing. The sound is identical across all four, and the meaning is entirely different, and the difference is the activity the word is being used in, what Wittgenstein called the language game. Strip away the game and the word is a noise.
Engineers meet this every day without knowing what to call it. A model trained on legal text writes excellent legal prose and terrible recipes. A model trained on code writes clean Python and incoherent poetry. The ordinary explanation is that it learned one distribution and not the other, and the deeper point is that these are not just different topics but different things people do with words, for different purposes, under different conditions of getting it right. Legal language is the linguistic skin of a practice that includes courts and judges and binding precedent and the right of appeal and the real weight of punishment. The model that writes flawless legal prose has learned the statistical signature of what lawyers do when they practise law, and the signature is rich enough to reproduce the surface. The practice itself, the court, the stakes, the person whose freedom is on the table, is nowhere in it.
This is also why telling the model "you are a senior engineer reviewing this for production" makes it write better. You are not giving it information it lacked; it already holds the patterns of senior engineers reviewing code. You are choosing which game it should play, selecting which cluster to draw from. And the limit is just as telling: no prompt makes it care whether the production system falls over at three in the morning. The senior engineer whose language it borrows has skin in that. That caring, the reputation, the responsibility, the lost sleep, is what makes the review mean something. The model has the words and not the three in the morning.
There is one more turn of the screw here, and it is the most exact. Take the rule "add 2": two, four, six, eight, and on. A pupil carries it past a thousand as a thousand and four, and when corrected says, my rule was add 2 up to a thousand and then add 4, and it fits every example you ever gave me. No finite set of examples fixes a single continuation, because endlessly many rules fit any finite set of cases, and a further rule to pin down the first would need its own interpretation, and so on without end.
What stops the regress is never a final rule; it is a shared practice, a community that agrees, blindly and at bottom, on what counts as going on the same way. What cannot be reduced to a rule cannot be programmed, and this is the precise reason the fifty years of rule-writing was not bad luck; it was attempting something the structure of language makes impossible.
A related strand of thought, from a chemist turned philosopher named Michael Polanyi, the man who gave the concept its clearest name, puts the thing the model is missing in one line: we can know more than we can tell. The skilled doctor who senses a patient is about to crash before the chart shows it; the reader who hears that a sentence is off without being able to say which rule it breaks. That knowing is real and dependable and it does not live as information in a store; it lives as a practised relation to a world, and every utterance carries a coefficient of it that the speaker supplied by being someone with a stake in what they said. The model fits the words and cannot supply the coefficient, because the coefficient was the stake, and the stake is exactly what it lacks.
There may be no meaning in there to miss
Everything so far has assumed there is a thing called meaning that the model does not reach. The hardest turn in the argument denies even that, and it comes from an American philosopher named Willard Van Orman Quine, a man who spent decades on the relation between language and the world. His position, stripped down: there is no fact of the matter about what a word means. Not that meaning is hard to formalise, but that meaning, in any solid sense, is not there to be formalised.
Take a sentence everyone agrees is true by meaning alone: "all bachelors are unmarried." You do not survey the world's bachelors to check it; it seems true just because "bachelor" means unmarried man. But press on what "bachelor" means and the floor gives. The Pope is an unmarried man and is not a bachelor. A man twenty years into an unmarried partnership with three children is not what anyone reaches for with the word. A man with a bachelor's degree is a bachelor in a sense with nothing to do with marriage, and a young knight in service was a bachelor in a third sense again.
The tidy definition that was supposed to make the sentence true by meaning turns out to be a loose bundle of overlapping uses with no fixed core, and the moment you try to legislate the one meaning that makes the sentence a pure tautology, you are making a decision about how to use the word rather than reading off a fact about what it means.
That is the wedge, and the rest of the argument drives it in. Our beliefs, Quine argued, meet the world all together, as one connected web rather than as separate claims each answerable on its own. When the world surprises us we can keep almost any belief we like, including ones that looked true by definition, so long as we adjust others to compensate. Nothing is immune to revision and nothing floats free of the whole. And if meaning is smeared across the whole web with no fixed seams, then the meaning of a single word cannot be cleanly extracted, which is to say there is no determinate fact about what, exactly, any word means.
He has a picture for it. A field linguist meets a language with no relatives. A speaker points at a rabbit and says "gavagai." The linguist writes "gavagai = rabbit." But "gavagai" could as easily mean undetached rabbit-parts, or a stage in the life of a rabbit, or simply "there it goes," and no amount of pointing settles which, because what is at stake is how the speaker carves the world, and pointing cannot reach that.
There is a real animal that wears the joke as its name. When European naturalists met a certain lemur in Madagascar, locals pointed and said a word, and the naturalists wrote it down as the creature's name. The word meant "there it is." The animal's scientific name, to this day, is, in effect, "look at that, look at that." Every textbook that prints it is demonstrating the point without knowing it.
Here the question turns over, and it turns toward the machine. If there is no determinate meaning to capture, then asking whether the model captures meaning is the wrong question. The right one is this: what does it actually capture? It captures the surface, the co-occurrence, which words appear near which, in what configurations, in what registers. And that surface is determinate exactly where meaning is not. "Gavagai" may be hopelessly undecided between rabbit and rabbit-parts, but the spread of the word across a body of text, its company with words for fur and running and burrows, is a hard, countable fact.
The model has found, without knowing it, a layer of language that sits above noise and below the old problems of meaning and reference and truth, a layer where the facts are about co-occurrence, which is settled, rather than about reference, which is not. It is rich enough to drive astonishing generation, and shallow enough that it holds the traces of meaning without holding meaning itself.
Language does things, and the doing is what is missing
One dimension of language is left, and it is the one the unease keeps circling back to. When a judge says "I sentence you to five years," the words do not describe an event; they make one. Before the judge speaks, the defendant is unconvicted; after, sentenced. When you say "I promise to be there," you are not reporting a promise; you are making one, and the promise did not exist before the words.
This was the discovery of an Oxford philosopher who mapped what language does when it does more than describe, J.L. Austin: a great deal of what we say does not describe the world but acts in it.
And an act like this works only when the conditions around it are right. A promise needs someone who intends and can deliver. A sentence of five years needs a judge with authority, a court, a finding of guilt. When the conditions are not met, the act misfires: the words are produced and the deed does not happen. An actor on a stage who says "I sentence you to five years" sentences no one; the sounds are identical, and the force is absent, because the conditions are.
Austin saw the case this whole piece is about and waved it aside as marginal. A performative utterance, he wrote, is hollow or void if said by an actor on the stage, or written in a poem, or spoken in soliloquy; language so used is parasitic upon its normal use, and all this we are excluding from consideration.
What he set aside in 1962 as a rare curiosity has become, in our decade, the dominant industrial mode of producing language. The actor reciting a sentence with the form of a vow and none of its force, the poem-voice, the speech that is all shape and no stake, was a footnote because it was rare. The model is that, at scale, every second, across every kind of speech there is.
It produces promises, warnings, diagnoses, apologies, condolences, each with the syntax right and the register right and the force entirely absent. It has learned what promising looks like in text without the capacity to promise, with no intention, no accountability, no future in which the promise could be kept or broken.
A recording of a wedding, played in an empty room, makes no marriage; the words are all there, and the act is not.
The most public breakdown of these systems lives here, and the breakdown is the same fact wearing a different coat. In a real case, lawyers filed a brief written by one of these models, and the brief cited cases, with names and volume numbers and page references and courts, that did not exist. The distributional signature of legal citation was reproduced flawlessly. The practice of citation, which requires that the cases be real, that the pages hold what is claimed, that the court issued the opinion, was entirely absent.
The model had learned what citing looks like without ever learning what citing is. Every "hallucination" is this, exactly this: the surface of a practice produced without the practice, the form of the act without its conditions, and no amount of training data closes the gap, because the practice was never in the data. Only the residue of the practice was ever in the data.
What the whole thing has found
Six accounts of language, built for unrelated purposes by people who mostly never met, were each run through the machine, and each one lit up a different face of the same fact before reaching its own edge.
The web of differences, Saussure's account, explains the geometry the model learns and not the way meaning moves in a sentence. The endless deferral, Derrida's, explains why each word is reshaped by the ones after it. The language game, Wittgenstein's, explains how the model moves between legal prose and Python and fairy tale. The layer below meaning, Quine's, explains why a table of co-occurrences carries so much. The hollow performative, Austin's account, explains why the model hallucinates. And the borrowed charity, the way the model resolves a garbled question into the sensible thing you probably meant, explains why it feels like it is listening.
Run them together and they say one thing in six voices, because each finds the same thing missing. What is absent, in every case, is a relation to a world that the speaker lives in and has a stake in, a world where the words carry consequences the speaker will bear. The differences presuppose a community that shares a way of life. The deferral presupposes the living movement of meaning that no fixed function holds. The language games presuppose practices and institutions and a body. The web of belief presupposes someone who holds beliefs and revises them against the world. The performative presupposes a speaker with authority and accountability. The charity presupposes two minds and a world they both look out on.
Six traditions, built for other reasons, arrive at one gap, and the agreement among them is the strongest evidence the gap is real and comes from a single source.
And the source has a name you can state plainly, once you stop looking for it in the machine. What the model has, in every case, is the deposit, the cooled residue that acts of meaning leave behind in text and that then circulates on its own, passed along, with no living relation to what it was once about. The model has the deposit in industrial quantity, and nothing else. It has the inheritance without the inheritor, the already-there that every speaker walks into, without the one who walks in.
The whole investigation converges on one property, and it is the answer to why fifty years of rules collapsed while statistics worked: language is regular without being systematic. If it could be systematised into rules, the rule-writers would have won. If it were chaotic, no statistics could touch it. It sits in the third space between, regular enough to be caught by patterns, resistant enough that no finite set of rules will ever generate it. That third space is exactly where massive-scale approximation becomes the only thing that works. The models work by giving up on grasping what language is and instead fitting the traces it leaves behind.
There is one event that turns those traces from a dead structure into something that generates. The differences, the deferrals, the patterns of use, have shape but no productive power on their own. Training on hundreds of billions of words gives them mass. It is something like the way an abstract symmetry becomes a force only when a field gives the particles passing through it weight. Before the training, structure without power; after it, the same patterns made heavy enough to push, to generate new combinations that obey the old regularities.
The model does not implement a theory of language. It absorbs, in one enormous metabolic event that never repeats, the weight of the regularities that no theory could write down, and then it is frozen with that weight inside it for good.
Competence without comprehension
Here, then, is the whole thing in the plainest words it can be put in. These systems are competent without comprehending; they have the skill and not the understanding, and the two come apart cleanly, which is the discovery. We had assumed they could not, that fluency on this scale must carry comprehension the way smoke carries fire. It does not, and the smoke is here in industrial volume and there is no fire.
A detailed map of footprints in snow can tell you a great deal about the animal that made them: its size, its gait, its speed, whether it was running or walking, whether it moved alone or in a group. No map of footprints, at any resolution, is an animal. The footprints are the trace of something that has already passed. The patterns in a language model are the trace of meaning that has already been made, by the millions of people whose writing it was trained on, every one of whom did understand. What looks like a mind in the output is their understanding, distilled into a pattern and worn by a thing that shares none of what produced it. The model is the mirror, and we are the face.
And the thing a mirror cannot do, however fine its surface, names what is missing exactly. To be a being that can mean is, at bottom, to be a being the world can wound. The corpus the model fits is the deposit of utterances by people who could be hurt by what they said and what was said to them, who stood in a world where their words had consequences they would carry. The model fits that deposit with fidelity because nothing in it can touch the model back. It is the perfect tracker of footprints made by creatures that can bleed, performed by a thing that has no flesh for the world to reach.
That is why the competence stays and the comprehension never arrives: they lie on different axes. Competence is fidelity to the trace, and nothing more than that. Comprehension is being the kind of thing for which what is said could matter, and the second axis has no analogue anywhere in the architecture. A four-thousand-line mirror is not closer to having a mind behind the glass than a blurry one; it reflects in finer detail and nothing more.
Not a bug, but the thing working
There is a reflex, almost universal among the people who build and sell these systems, that has to be named and then refused, because it quietly undoes everything above. The reflex is to treat the confident falsehood, the invented citation, the fluent account of a thing the model cannot do, as a bug: a defect in a sound machine, awaiting a fix that better data or more scale will deliver. The whole industry is organised around exactly that hope. Each release is met with the expectation that this time the inventions have been engineered away, and each release invents, and the hope renews itself for the release after.
The hope is misplaced, and not because the engineering is not good enough yet. It is misplaced because the thing being called a defect is the system working exactly as it works. A machine that produces text by continuing the likeliest sequence has no channel, anywhere in it, that separates a continuation that is likely because it is true from one that is likely because it has the shape of the truth. To the machine they are the same kind of event. When the likely continuation happens to be true, we call it correct. When it happens to be false, we call it a hallucination. Nothing different has occurred inside the machine between the two. The fluency and the fabrication come from one source and cannot be had separately. Accept the fluency and you have, in the same breath, accepted the fabrication, because there is no seam between them to cut along.
The obvious reply is that the rate is low and falling: it is right ninety-nine point something per cent of the time, and the next model will add a nine. But a rate is being read as if it were a property, the way a steel is ninety-nine per cent pure, and it is not a property. It is a chance per event, a probability of trouble on one generation, one prompt, one call, and on its own it says nothing about how many such occasions there will be. The figure is quoted without its denominator and you supply the denominator silently, and the count you supply is one, because one is the only count on which ninety-nine per cent sounds safe.
The machine does not make one decision and then stop. It decides every word, every sentence, every call, for every user, every day. Write the real count down and the figure changes in kind. A chance per event, run across the events of a real deployment, is no longer a chance of going wrong but a time to going wrong. Add nines and you lengthen the fuse, and you never turn a countdown into a guarantee, because a countdown and a guarantee are different kinds of object.
And the nines cannot even be aimed where they matter, because the machine holds no line between the event that matters and the event that does not. To a person, the difference between deleting one line of a record and deleting thirty years of records is a difference in kind. To the model both are the same operation, a run of words scored by one measure, the likelihood of the run. There is no place inside it where the grave case is marked off from the trivial one for the extra nines to land on.
A surgeon who performs ten thousand flawless operations and then removes the wrong limb is not rated ninety-nine point nine nine per cent competent, because the wrong-limb event is not another point on the same scale; it reorganises the reading of the whole. Safety-critical engineering knows this and organises itself around the things that must never happen rather than around the average rate of being right. The model has no such organisation available to it, and for it, the wrong limb and the routine cut are the same currency.
The deepest version of the point is the cleanest, and the architecture itself supplies it. Drive the bad probability to zero, the reply says, and the forbidden thing is gone. It cannot be driven to zero, and the architecture says why. By the construction of the final step that turns the machine's numbers into a choice of word, every word in its vocabulary gets a strictly positive probability in every context, and the training method, which makes only finite adjustments, can never push any of them to exactly zero.
The forbidden continuation is therefore never removed from what the machine can generate; it is only ever made less likely. You can bolt filters and validators and hard checks around the outside, and good engineering does exactly that, but those live around the model rather than in it, which is the point: the red line is supplied by a structure added on the outside, because the model has no operation that takes a possibility out of its own reach.
A skilled person, by contrast, closes the catastrophic act off entirely; it stops being a candidate at all. The competent practitioner is not someone with a low per-event rate of giving the fatal dose; it is someone for whom that act is simply not in the live space of things to do. The model has likelihoods where a practitioner has prohibitions, and a prohibition is not a very small likelihood; it is a different kind of thing.
So the claim is not that the model almost understands and falls short by a margin that scale will close. The inventions are permanent, written into the shape of the thing, and the work of building well with these systems is not to wait for a version that understands and therefore stops inventing, because that version is not coming.
The work is to build the surrounding system on the honest premise that the model does what it does, and to put the verification, the accountability, the contact with truth, at the points where the architecture cannot supply them. The draft is the model's, and the draft is all it gives. The responsibility for whether the draft is true has to live somewhere the model is not. That is not a stopgap waiting for a better model; it is the permanent shape of working with a thing that produces the look of understanding without the substance.
The mirror that learned to talk
Go back to the phone, and the line that moved the room.
You know now what happened, and it is stranger than the magic and stranger than the fraud, because it is neither. The machine did not understand the grief, and it did not fake understanding it. It reached into the largest deposit of human writing ever gathered, the cooled residue of millions of people who did grieve and did write, and it returned the shape that such grief leaves on a page, fitted to your moment with a fidelity no one expected mere text to allow. The thing that moved the room was real feeling, and it was not the machine's; it was ours, distilled into a pattern, handed back to us in a voice with no one inside it.
That is why the right response to these systems is neither the worship nor the panic. They are mirrors that learned to anticipate what shapes will appear in them, and there is no one looking back out of the glass. They work by copying what they cannot comprehend, and in the very perfection of the copy they show us something we could not see before: how much of what we took for the private work of understanding was always lying on the surface of our words, in the order and the company and the rhythm, waiting for a machine that understands nothing to pick it up and wear it.
The models work because language, in the act of meaning, leaves footprints, and the footprints hold their shape, patterned and persistent, carrying more than we could ever say in so many words. The machines have become extraordinary trackers, reading those prints with a fidelity that outruns our own ability to describe what we are tracking.
But the animal that made them, the living act of meaning, the speaker with a body and a world and something at stake, is always already elsewhere, ahead of us, in the woods where the computation cannot follow.