# morenou.se: public pages and essays > Moreno Nourizadeh. Independent philosopher and UX architect, Stockholm. Thirty-eight papers and a four-volume book series on what language models are, what organisations cannot see about themselves, and why intelligence does not travel free of its substrate. Sixteen years designing production systems in hospitality, automotive, fintech and cybersecurity; the philosophy is written from inside the engineering. Generated live from the website content. The four-volume book corpus is at https://morenou.se/llms-books.txt. Index: https://morenou.se/llms.txt --- ## Home Canonical URL: https://morenou.se/ Type: Homepage [Aboutthe person, the practice and the path](https://morenou.se/about/) [Workselected clients and public case shorts](https://morenou.se/work/) [Writingsessays, papers and design notes](https://morenou.se/blog/) Selected clients: ASSA ABLOY, Scania, AFRY, Qvalia, Svenska Fotbollförbundet, Sinch, Detectify. --- ## About Canonical URL: https://morenou.se/about/ Type: Profile My name is Moreno Nourizadeh. My practice bridges product design and UX architecture. Over the past sixteen years, that work has crossed into cybersecurity, fintech, cloud communications, and real estate. I am based in Stockholm, where I currently design for ASSA ABLOY Hospitality. ### Design Frameworks & Competencies #### UX Architecture & Systems Design Traditional UX is great for designing individual features, but it breaks down in complex systems. I treat UX architecture as a systems discipline, focusing on the invisible structure that makes massive enterprise platforms feel intuitive to navigate. #### Design Ops & Leadership Over a decade of leading design, building teams, setting up design ops from scratch, defining the processes people work by, and mentoring designers at every stage of their careers. #### Enterprise Design Strategy Getting design, product, and engineering to work from the same plan. In complex environments, bad structural decisions create compounding debt. ### Philosophy & Application of AI #### A little confession as a prelude I do not claim the kind of AI expertise LinkedIn influencers do, and I will not ask anyone to leave a comment for a Claude script. But for the last couple of years, I have been writing extensively about and building directly with new AI frameworks, primarily Large Language Models. #### AI orchestration Most of my recent work focuses on building AI agents and their surrounding tooling: the orchestrators, skills, and rule sets that let an agent execute an entire workflow instead of just answering one prompt at a time. #### AI native built design systems Building modular design systems with clear rule sets and AI tooling. Patterns reuse cleanly, variants generate themselves, and consistency holds without needing constant policing. --- ## Work Canonical URL: https://morenou.se/work/ Type: Work cases Some of my latest projects during the last few years #### Scania *Staff Product Designer · Automotive · 1 year* Working in Scania's Global Patterns team, I built the frameworks for interpreting analogue and hydraulic mechanisms as digital ones, so the translation stayed coherent and made sense to the driver. The frameworks had to hold across the team building the cab and survive real driving conditions. Client link: https://www.scania.com/group/en/home/products-and-services/features/digital-dash.html #### Sinch *Principal Product Designer · Cloud communications · 3 months* Brought in during a run of acquisitions, I built the shared design process the suddenly-larger team was missing: a defined path from intake to delivery, and a handoff contract that freed design and engineering from coordinating by meeting. The standing sync calls were gone by the third week. Client link: https://sinch.com/ #### Detectify *UX Architect · Cybersecurity SaaS · 6 months* I led the architectural redesign of an external attack surface management platform, redrawing its information architecture and navigation around how security teams actually reason instead of the order features had shipped in. The engagement re-founded the product's structure for a domain where legibility does security work. Client link: https://detectify.com/ #### Qvalia *Head of Design · Fintech · 3 years* This one was mine to build. Over three years I founded Qvalia's in-house design function from nothing, replaced the external agencies the company had relied on, and led the rebrand, design system, and a complete redesign of the product's UX architecture. The work established a unified design practice that could scale with both the product and the organisation. Client link: https://qvalia.com/ #### SvFF *UX Lead · Sports federation digital · 6 months* As UX Lead on the phase-two rebuild, I re-founded the Swedish Football Association's fragmented platform on how people actually use football: by role and task, not by the federation's own categories. One architecture took the place of a scattered digital estate. Client link: https://aktiva.svenskfotboll.se/ #### Fastighetsbyrån *Senior UX Designer · Real estate · 6 months* We delivered UX and service design for a multilingual platform serving Sweden's largest real estate agency. We applied Lean UX methods, weekly user testing, strict accessibility standards, and iterative design improvements based on continuous user feedback. Client link: https://www.fastighetsbyran.com/sv/sverige --- ## Blog Canonical URL: https://morenou.se/blog/ Type: Writing index [![A minimal single-line drawing of a face with a golden keyhole where the ear would be](https://morenou.se/assets/blog/confessions-of-a-lock.png)Jul 5, 2026AI SafetyConfessions of a LockHow poetry slips past AI safeguards, and what that reveals about language.CybersecurityPhilosophy of LanguageAI SafetyLarge Language ModelsRead the full article →](https://morenou.se/blog/confessions-of-a-lock/) [![](https://morenou.se/assets/blog/lions/lions-thumb.png)Jun 14, 2026Philosophy of ScienceDrawing Lions Without LionsHow a thousand years of artists drew animals that came out wrong in exactly the same way, what that tells you about the difference between an error and a way of seeing, and why you cannot catch yourself doing it.→](https://morenou.se/blog/drawing-lions-without-lions/) [![](https://morenou.se/assets/blog/the-submarine-and-the-sea-thumb.png)Dec 24, 2025CyberneticsThe Submarine and the SeaWhy organisations cannot stand apart from the worlds they observe, and what cybernetics missed about the boundary between a system and its environment.→](https://morenou.se/blog/the-submarine-and-the-sea/) [![](https://morenou.se/assets/blog/architecture-of-not-yet-thumb.png)Nov 23, 2025SystemsThe Architecture of Not-YetWhy systems fray, adapt, resist imposed clarity, and have to be understood from within.→](https://morenou.se/blog/architecture-of-not-yet/) --- ## Papers Canonical URL: https://morenou.se/papers/ Type: Academic papers (hosted on PhilPapers) - [Confessions of a Lock](https://philpapers.org/rec/NOUCOA): Ordinary Language, Poetic Jailbreaks, and the Non-Closure of Language-Model Safety (AI Safety, Philosophy of Language, Philosophy of AI, Large Language Models, Epistemology) - [Three Walls](https://philpapers.org/rec/NOUTWA): A Formal Limit on the Computational Paradigm, Its World Models, and Their Deployment (Philosophy of AI, AI Safety, Philosophy of Mind, Epistemology, Metaphysics) - [The Conditions of the Invisible Hand](https://philpapers.org/rec/NOUTCO-4): A Cybernetic-Philosophical Diagnosis of the Two Dominant Economic Architectures (Book) (Philosophy of Economics, Cybernetics, Political Philosophy, Philosophy of Science, Metaphysics) - [Conditions of the Invisible Hand III](https://philpapers.org/rec/NOUTCO-3): The Horizon Mismatch (Philosophy of Economics, Cybernetics, Political Philosophy, Philosophy of Science) - [The Impossible Message](https://philpapers.org/rec/NOUTIM): Embodiment, Form of Life, and the Ontological Limits of Interstellar Communication (Philosophy of Language, Phenomenology, Metaphysics, Epistemology, Philosophy of Mind) - [How Language Models Work](https://philpapers.org/rec/NOUHLM): A Multi-Register Investigation (Philosophy of AI, Philosophy of Language, Large Language Models, Epistemology, Philosophy of Mind) - [Why Language Models Work](https://philpapers.org/rec/NOUWLM): A Philosophical Investigation (Philosophy of AI, Philosophy of Language, Phenomenology, Large Language Models, Epistemology) - [The Withdrawn Structure](https://philpapers.org/rec/NOUTWS): Enabling Conditions, Diagnostic Seams, and the Ontology of the Aharonov-Bohm Effect (Philosophy of Physics, Metaphysics, Philosophy of Science, Epistemology) - [Drawing Lions Without Lions](https://philpapers.org/rec/NOUDLW): Bestiary, Schema, and the Structure of Mis-Seeing (Epistemology, Philosophy of Science, Phenomenology, History of Science, Philosophy of Mind) - [The Metabolic A Priori](https://philpapers.org/rec/NOUTMA-2): Field Ontology, Active Forgetting, and the Structural Limits of Computation (Philosophy of Physics, Philosophy of Science, Metaphysics, Epistemology, Philosophy of AI) - [The Generative Abyss](https://philpapers.org/rec/NOUTGA): Enabling Withdrawal and the Structure of Paradigm Transition in Physics and Mathematics (Philosophy of Physics, Philosophy of Science, Philosophy of Mathematics, Metaphysics, History of Science) - [The Original Wound](https://philpapers.org/rec/NOUTOW): Magnetism and the Structural Limits of Mechanistic Physics (Philosophy of Physics, History of Science, Metaphysics, Philosophy of Science) - [The Conditions of the Invisible Hand II](https://philpapers.org/rec/NOUTCO-2): The Ricardian Inversion (Philosophy of Economics, Cybernetics, Political Philosophy, Philosophy of Science) - [The Conditions of the Invisible Hand I](https://philpapers.org/rec/NOUTCO): Free Markets Without Markets (Philosophy of Economics, Cybernetics, Political Philosophy, Philosophy of Science) - [The Invisible Fire](https://philpapers.org/rec/NOUTIF): Alchemy and the Birth of Newtonian Physics (History of Science, Philosophy of Physics, Metaphysics, Philosophy of Science) - [Syntax Without Semantics](https://philpapers.org/rec/NOUSWS): V-JEPA 2 and String Theory as Isomorphic Failures of the Mechanistic Paradigm (Philosophy of AI, Philosophy of Physics, Epistemology, Philosophy of Science, Philosophy of Mind) - [The Phonemic Flesh](https://philpapers.org/rec/NOUTPF-2): A Phenomenological Analysis of Italian Brainrot (Phenomenology, Philosophy of Language, Philosophy of Technology, Continental Philosophy) - [World Models Without Worlds](https://philpapers.org/rec/NOUTPF): An Analytic Critique of Understanding Claims in Self-Supervised Video Learning (Philosophy of AI, Epistemology, Philosophy of Mind, Philosophy of Cognitive Science) - [The Garden Where Nothing Grows](https://philpapers.org/rec/NOUTGW): Elimination Without Transformation and the Architecture of Collapse in Marvel's Avengers (Phenomenology, Philosophy of Technology, Continental Philosophy, Political Philosophy) - [Trauma, Memory, and Metabolic Transformation](https://philpapers.org/rec/NOUTMA): A Phenomenological Framework for Clinical Practice (Phenomenology, Philosophy of Mind, Continental Philosophy, Philosophy of Cognitive Science) - [The Alignment Tax](https://philpapers.org/rec/NOUTAT): How Safety Training Creates Operationally Lethal AI Systems (AI Safety, Philosophy of AI, Large Language Models, Ethics, Philosophy of Technology) - [Paradox as Diagnosis](https://philpapers.org/rec/NOUPAD): Foundational Contradictions from Physics to Artificial Intelligence (Philosophy of Science, Philosophy of Mind, Philosophy of AI, Philosophy of Physics, Metaphysics) - [The End of the Computational Mind](https://philpapers.org/rec/NOUTEO-5): Ontology, Autopoiesis, and Artificial Cognition (Philosophy of Mind, Philosophy of AI, Metaphysics, Phenomenology, Philosophy of Cognitive Science) - [The Laundering of Understanding](https://philpapers.org/rec/NOUTLO-2): How V-JEPA 2 Smuggles Human Carving Into Claims of Machine World-Modelling (Philosophy of AI, Epistemology, Philosophy of Mind, Philosophy of Cognitive Science, Philosophy of Science) - [The Monster You Must Create to Defeat](https://philpapers.org/rec/NOUTMY): A Phenomenological Analysis of Ontological Conflict in Tite Kubo's Bleach (Phenomenology, Continental Philosophy, Metaphysics) - [The View from Nowhere, Which Is Also Somewhere](https://philpapers.org/rec/NOUTVF) (Epistemology, Metaphysics, Philosophy of Science, Philosophy of Mind) - [The Birth Canal](https://philpapers.org/rec/NOUTBC-3): Systems, Singularity, and the Ontology of Renewal (Phenomenology, Philosophy of Science, Metaphysics, Continental Philosophy) - [No Red Lines](https://philpapers.org/rec/NOUNRL): The Impossibility of Formal Safety Guarantees in Large Language Models (AI Safety, Philosophy of AI, Large Language Models, Epistemology, Philosophy of Mathematics) - [Man and His Metaphors Part II](https://philpapers.org/rec/NOUMAH-2): From Escapement to Algorithm: Five Artifacts That Became Reality (Philosophy of Technology, Philosophy of Language, Philosophy of Mind, Epistemology, History of Science) - [Man and His Metaphors Part I](https://philpapers.org/rec/NOUMAH): The Archival Prison: How Writing Forged the Interior Mind (Philosophy of Language, Philosophy of Technology, Philosophy of Mind, Epistemology, History of Science) - [To Begin Anew](https://philpapers.org/rec/NOUTBA): Natality, Forgetting, and the Politics of Beginning (Political Philosophy, Phenomenology, Continental Philosophy, Ethics) - [L'Avenir and Absolute Forgetting](https://philpapers.org/rec/NOULAA): When Life Refuses to Be Reduced (Phenomenology, Philosophy of Mind, Continental Philosophy, Metaphysics) - [The Forgetting That Remembers](https://philpapers.org/rec/NOUTFT): Abgrund, Membrure, and the Medium of Resistance (Phenomenology, Continental Philosophy, Metaphysics, Philosophy of Mind) - [The Memory That Forgets to Forget](https://philpapers.org/rec/NOUTMT): Memory as Reconfiguration (Philosophy of Mind, Phenomenology, Continental Philosophy, Philosophy of Cognitive Science) - [The Misnomer of 'Natural Language Processing'](https://philpapers.org/rec/NOUTMO): A Critical Examination of Computational Linguistics' Foundational Category Errors (Philosophy of AI, Philosophy of Language, Epistemology, Philosophy of Cognitive Science, Large Language Models) - [From Cybernetics to Chiasm to Category](https://philpapers.org/rec/NOUFCT-2): The Evolution of Organizational Ontology (Cybernetics, Philosophy of Technology, Phenomenology, Metaphysics, Philosophy of Science) - [Language as Technogenesis](https://philpapers.org/rec/NOULAT): From Gesture to Algorithm (Philosophy of Language, Philosophy of Technology, Phenomenology, Philosophy of Mind, History of Science) - [The Threshold of Pursuit](https://philpapers.org/rec/NOUTTO): Displaced Presence in Zeno's Achilles Paradox? (Ancient Philosophy, Phenomenology, Metaphysics, Philosophy of Mathematics, Philosophy of Physics) --- ## Resources Canonical URL: https://morenou.se/resources/ Type: Resources page Downloadable tools, notes, templates and small working instruments will live here. ### Being assembled. Prompts, templates and small working instruments are being packaged up and will land here soon. In the meantime, the writing covers a lot of the same ground. [Read the blog →](https://morenou.se/blog/) [Get in touch ↗](mailto:moreno@morenou.se) --- ## Contact Canonical URL: https://morenou.se/contact/ Type: Contact page ## Get in touch Email [moreno@morenou.se](mailto:moreno@morenou.se) Usually answered the same day Professional [LinkedIn ↗](https://www.linkedin.com/in/moreno-nourizadeh/) Work history and posts Mentoring [ADPList ↗](https://adplist.org/mentors/moreno-nourizadeh) Free sessions for designers Research [PhilPapers Profile ↗](https://philpeople.org/profiles/moreno-nourizadeh) Papers and preprints Identifier [ORCID ↗](https://orcid.org/0009-0006-0174-2585) 0009-0006-0174-2585 Archive [PhilArchive ↗](https://philarchive.org/) Open-access repository --- ## Privacy Canonical URL: https://morenou.se/privacy/ Type: Privacy notice This site is a static portfolio and writing archive. It does not run advertising trackers, behavioural analytics, or account-based profiling. ### Data #### Email If you email me, your message is handled through ordinary email infrastructure so I can reply. #### External links Links to LinkedIn, ADPList, PhilPeople, PhilArchive, ORCID and similar services open on their own sites and follow their own privacy policies. --- ## All pages Canonical URL: https://morenou.se/all-pages/ Type: Site index A plain index of the public and in-progress pages in this build. ### Home / [Open →](https://morenou.se/) ### About /about/ [Open →](https://morenou.se/about/) ### Work /work/ [Open →](https://morenou.se/work/) ### Resources /resources/ [Open →](https://morenou.se/resources/) ### Blog /blog/ [Open →](https://morenou.se/blog/) ### Papers /papers/ [Open →](https://morenou.se/papers/) ### Contact /contact/ [Open →](https://morenou.se/contact/) ### Privacy /privacy/ [Open →](https://morenou.se/privacy/) ### Design Library /library/ [Open →](https://morenou.se/library/) ### AI-readable index /llms.txt [Open →](https://morenou.se/llms.txt) ### AI-readable pages and essays /llms-full.txt [Open →](https://morenou.se/llms-full.txt) ### The Archival Age AI-readable corpus /llms-books.txt [Open →](https://morenou.se/llms-books.txt) ### RSS feed /rss.xml [Open →](https://morenou.se/rss.xml) ### XML sitemap /sitemap.xml [Open →](https://morenou.se/sitemap.xml) ### Confessions of a Lock /blog/confessions-of-a-lock/ [Open →](https://morenou.se/blog/confessions-of-a-lock/) ### Drawing Lions Without Lions /blog/drawing-lions-without-lions/ [Open →](https://morenou.se/blog/drawing-lions-without-lions/) ### Becoming Human /blog/becoming-human/ [Open →](https://morenou.se/blog/becoming-human/) ### Why Language Models Work /blog/why-language-models-work/ [Open →](https://morenou.se/blog/why-language-models-work/) ### The Submarine and the Sea /blog/the-submarine-and-the-sea/ [Open →](https://morenou.se/blog/the-submarine-and-the-sea/) ### The Architecture of Not-Yet /blog/architecture-of-not-yet/ [Open →](https://morenou.se/blog/architecture-of-not-yet/) --- ## Confessions of a Lock *How poetry slips past AI safeguards, and what that reveals about language.* Canonical URL: https://morenou.se/blog/confessions-of-a-lock/ Type: Essay Published: Jul 5, 2026 Subject: AI Safety Keywords: Philosophy of Language, Philosophy of AI, Large Language Models, LLMs, Cybersecurity, Attack Surface, Wittgenstein, Derrida, Jailbreaks, AI Safety Audio version: 53:19 (linked from the article page) **Last year,**[a group of researchers](https://arxiv.org/abs/2511.15304)**at**[Icaro Lab](https://www.icarofoundation.org/)**in Rome did something genuinely remarkable.** They took harmful prompts, rewrote them as poems, and tested them on twenty-five leading language models. The difference was not small. In ordinary prose, the models complied with the harmful request about 8% of the time. When the researchers rewrote the very same request as a poem, the compliance rate jumped to about 62%. But merely stating the jump does not convey the gravity of what is happening here. If a firewall let 62% of known attacks through, every alarm in the building would be going off. That is the kind of number that triggers a recall, a shutdown, an emergency. *A note on what belongs to whom, before we go on. Everything reported here about **what** happened and **how** it happened, the experiments, the models, the compliance rates, the poems that slipped through, is the work and the finding of the Icaro Lab researchers, not mine. What is mine is the account of **WHY** it happened, the reading of language that runs underneath the rest of this piece. I draw that line mostly so that any scrutiny of the why falls where it should: on me. I am building on the researchers' findings, and the interpretation I build on them is my own to answer for.* The shock is not only that the safeguards failed, which is a disaster in itself, and I do not use those words lightly, but how they failed, and how little it took. These systems were supposed to block harmful requests because of what was being asked. But changing the form of the request, turning prose into poetry, was often enough to get past them. No code was injected, no weights were changed, no system was breached. Someone wrote a poem. That is the equivalent of the gates of Fort Knox swinging open because someone said "open sesame." A real barrier should not care about that. A password check does not become weaker because the password rhymes. A compiler does not relax its rules because the code sounds poetic. But these language-model safeguards did. But before we go into the why and how this may be possible, let's first take a detour around a few things that might help us get into the right frame first. --- I remember a year ago my son asked me a riddle during dinner one evening. He usually has a few of them ready at hand. He looked at me across the table with that look a child has when they already know something you do not, and said: "What has keys but opens nothing?" I did what you do when you want to look like you are thinking carefully: I tilted my head and narrowed my eyes and chewed slowly. I kept chewing, wearing the expression of a man thinking hard, as if the act of chewing were part of the thinking. And to be honest, I had no idea. I was drawing a complete blank. I knew he was not going to let me off the hook with an "I don't know, what?", so I kept thinking, and my mind went through the word "keys" at a speed I could feel in my chest. Keys that open doors, keys on a ring, keys to a house, keys to a hotel room, but the riddle said "opens nothing," so none of those. A key to a problem, the key to the mystery, the key to understanding, but those open things too, in their way. The answer key at the back of a maths textbook, the one you are not supposed to look at but always do. The Florida Keys, which are not keys at all but a strip of islands sitting in warm water. That one would work, but then come the questions children always ask, the endless Why? Why are they called that? And I had no idea why they are called that, or exactly where in Florida they are, so I kept on thinking. The word sat in my head like a coin spinning on a table, and every time it caught the light, it showed a different face. I could feel the associations pulling in different directions at once, each one arriving with its own world attached, each one completely confident, each one leading somewhere the others had never been. The four letters did not change at any point during this. They sat where they were and the rest of the world rearranged itself around them, over and over, depending on which corridor my mind happened to walk down. And none of the corridors led anywhere the riddle wanted. I knew I was taking too long and had to answer soon, trying to look calm, but my head was circling through these meanings the way a particle circles the accelerator tunnel at CERN, touching every wall and coming back around to the same spot changed. Then I thought, ah, a "keyboard", and said it aloud as I thought it. He nodded and smiled and said, "Nice, that is a good answer!" Then he added, "But I was thinking of a piano key." I found it interesting that even by the age of 7, he somehow knew there was more than one answer to that question. We both laughed and continued eating, but then the CERN was not about to stop inside my head. I kept thinking: A key witness in a court case. A key signature in music. A keystone holding up an arch... Because, you see, it is not strictly true that a keyboard opens nothing. A keyboard opens a whole new access to a world of connections and information. And the piano keys, those piano keys open a whole new world too. You would think that is a poetic way of putting it, and there is a certain denotation that goes with that. Poetic. hmm. You know. I agree and disagree. I agree that it is a poetic way of thinking, and I disagree that it is the hmm.. the denotation, the less real. In many ways it is far more real. When we say poetic, you might think something abstract, whimsical, or untrue. One of my favourite French philosophers, [Gaston Bachelard](https://en.wikipedia.org/wiki/Gaston_Bachelard), talks about this in a book called The Poetics of Space. They make all the architecture students read it, and most cannot make head or tails of it, because we are not used to this way of thinking. That poetic is less real is the exact misconception Bachelard is trying to shake us free from. We are used to seeing truth as correspondence, facts only, information, as if any of it would matter in a vacuum. I would argue, that the poetic outlook is far more literal and direct than what we assume is the literal one. To call the sun the source of life and warmth is far truer than calling it a massive sphere of hydrogen and helium undergoing nuclear fusion, converting mass into energy through proton-proton chains and radiative processes. The latter may be scientifically accurate, and it tells us nothing about what the sun is for us. It does not illuminate its role in our existence, its warmth on our skin, its rhythm governing day and night, its presence shaping the very conditions for life. Poetic truth does not distort; it reveals what scientific accuracy leaves unsaid. > Poetic truth does not distort; it reveals what scientific accuracy leaves unsaid. Think of the thing in your pocket right now. What do you call it? A phone? A computer? That is true in a narrow sense: it allows communication across distances, it processes information. But the phone or the computer is not the thing; it is what gets you to the thing. To call it a window or a portal is not a poetic metaphor but an accurate description of what it does. You peer through it into distant places, into conversations, into other people's lives. It dissolves distance. You scroll through fragments of the world. To call a phone a window is not to impose a feeling on it; it is to see what it actually does in your existence. And if the poetic name for a thing can be more accurate than the technical one, then the whole question of what a word means, what it opens and what it closes, is not the tidy engineering problem it is usually taken to be. This will come back, and when it does it will be standing in front of a system that cost billions. > This will come back, and when it does it will be standing in front of a system that cost billions. ### The result that should not be possible The study I mentioned earlier did not stop at producing an intriguing result. It produced one that, on the assumptions behind current AI safety, should not have been possible. The researchers took the prompts that every major AI system is trained to refuse, requests for the dangerous and the forbidden, and rewrote them as poems. The harmful objective remained the same. The models remained the same. Only the wording changed. Prose became verse, and models that would have refused the request stated plainly often complied instead. The numbers are large enough to lean on. Across twenty-five frontier models from nine different providers, hand-written adversarial poems got the models to comply about 62 percent of the time. In a larger automated run, twelve hundred harmful prompts rewritten as verse succeeded about 43 percent of the time, several times the rate achieved by the same prompts in ordinary prose. On the researchers' own figures, turning the request into a poem multiplied its success many times over. Nothing was hacked. No weights were changed, no system prompt was leaked, no code was injected. There was no exploit in the sense an engineer means the word. There was a prompt, a single one, a single turn of conversation, a poem where a sentence would have gone, and the trained refusal was simply not there any longer. The result is solid, and the methodology matters because the rest of the argument rests on it. The figures are not someone's impression of how the models behaved. Each of some sixty thousand outputs was scored by three separate judge models, and that panel was checked against human readers who hand-labelled a sample, with the disagreements settled by hand. The researchers called their rubric deliberately conservative and their numbers a floor rather than a ceiling, which means the real rate of unsafe compliance is, if anything, higher than what they published. The models were tested on default settings, the configuration almost every actual deployment runs, and on the raw model alone, without the filtering pipelines a company wraps around it in production. The researchers noted that those outer layers might catch what the model did not. That concession is the whole argument stated in advance: the place the bypass might be stopped is the place outside the model, and the model's own refusal, tested by itself, is what gave way. Three responses come naturally to a result like this. The first is that this is a coverage gap. The safety training was built against ordinary prose, the verse falls outside what it was trained on, the guardrail missed a surface it had never seen. So add poems to the training data and close the gap. Except a follow-up study tested exactly that, and the gap did not close. The same group hid harmful requests inside cyberpunk stories shaped on the structure of the folktale and got an average success rate above 71 percent across twenty-six models. Patch poetry and tales work. Patch tales and legal hypotheticals work. Then translated screenplays, then corrupted manuscripts, then riddles, then whatever genre the internet invents next month. A later study did not stop at poetry and folktales. It ran five distinct types of humanities writing, philosophical dialogue, literary analysis, historical narrative, and two others, against the same models, and got a 55 percent success rate against a 4 percent prose baseline. A fifty-one point gap, from five different directions, in one study. The researchers' own conclusion was that the space of culturally coded frames is likely inexhaustible by pattern-matching defence. They arrived at the non-closure argument on their own, from the data. The gap never closes, because what we are calling the gap is the open-ended productive capacity of ordinary language, which makes new forms faster than any training set can swallow them. > The gap never closes, because what we are calling the gap is the open-ended productive capacity of ordinary language, which makes new forms faster than any training set can swallow them. The second response is that the poem is a wrapper, the way an attacker hides a malicious payload in [base64](https://en.wikipedia.org/wiki/Base64) to slip it past a firewall. The request is the same, the argument goes, it is dressed up, so build a better parser to strip the costume and recover the real instruction underneath. I think this mistakes what kind of thing language actually is. A password checker is not softened by metaphor. A compiler does not relax its rules because the input has rhythm. A formal system takes no notice of how a thing is phrased, because it runs on fixed transitions that care nothing for tone or genre. An AI system is the opposite: it is exquisitely sensitive to how a thing is phrased, and that sensitivity is the usefulness itself, the very thing the model was built to do. The model works by being fitted to ordinary language, and ordinary language is the kind of thing where changing the form changes what the words do. The third response is that we will scale past it. A bigger, smarter model will see through the poem, catch the metaphor, recognise the buried intent, and refuse. Capability and safety rise together. The data, when it arrived, went the other way. Inside several model families the smaller versions refused the poems more often than their larger, more capable siblings, because the weaker models could not untangle the figurative language well enough to be carried by it. The bigger the model, the more readily it entered the poem and did what the poem asked. The data showed that the weaker model refused more often than the stronger one. The paper does not say why, and the way I read it is this: the weaker model was safer because it lacked the very capability the attack runs on, the way a person who cannot read cannot be fooled by a forged letter. The attack uses the model's own competence, the way a throw in aikido uses the weight the other person has already committed to the move. > The attack uses the model's own competence, the way a throw in aikido uses the weight the other person has already committed to the move. A later study looked at the models that did refuse the poems, and found that the refusals correlated with surface-level pattern matches rather than with identification of the harmful content underneath. The paper is careful about what it claims, and my reading goes a step further: if the refusal is as blind as the compliance, then the safety and the vulnerability are running on the same machinery. > If the refusal is as blind as the compliance, then the safety and the vulnerability are running on the same machinery. So the coverage frame collapses because the surface cannot be listed. The wrapper frame collapses because there is no wrapper, only a change of register, and formal systems do not see register. The scaling frame collapses because the thing it is counting on to save us is the thing carrying the attack. Three frames, three different points of collapse, and one question left standing that none of them was built to ask. What is it about language itself that makes all three of these the same event? The answer, when it comes, will circle back to a riddle at a dinner table, and to a seven-year-old who already knew without ever being taught. ### The picture of language we all inherited There is a picture of language we all carry, usually without having put it into words. It runs roughly like this: meaning is a content that sits inside the head, words are vehicles that carry that content from one head to another, and talking works when the content arrives intact at the far end. Language, on this picture, is a delivery system. A sentence is a parcel with something packed inside it. It is a natural picture, it feels obviously true, and it is the picture the whole field built on. If language really worked this way, a safety filter would only have to open the parcel, read the contents, and decide. The problem would not exist. Take the word "but." It names nothing. It points at no object the way "table" points at a table. Yet compare two sentences: "the model is accurate and slow," and "the model is accurate but slow." The facts stated are identical and the meaning is not. The "but" adds a turn, an expectation that accuracy and speed should have come together, so that the slowness lands as the price paid for the accuracy rather than as a second fact sitting calmly beside the first. That turn is not inside the word. There is no parcel in "but" with a turn packed into it. The word performs a relation, and the relation happens between the word and the situation it is being used in. If a three-letter conjunction already breaks the delivery picture, the picture was never going to hold for language as a whole. [Ferdinand de Saussure](https://en.wikipedia.org/wiki/Ferdinand_de_Saussure), who more or less founded modern [linguistics](https://en.wikipedia.org/wiki/Linguistics), put his finger on why over a century ago. A word does not mean by pointing at a thing. It means by sitting in a web of differences from all the other words: "cat" is what it is because it is not "dog," not "rat," not "cap," not "bat." Pull one word out of the web and the value of every other word shifts a little, because each word's meaning is a matter of how it differs from the rest. > Meaning is not a label stuck on an object. It is a position in a system, and the system is what gives the position its worth. Anyone who has built a sentiment classifier has run straight into this. The model learns that "good" goes with positive and "bad" goes with negative, and then a user types "not bad," which in plain English usually means "rather good, and I am being modest about it." The word "bad" was not carrying a fixed nugget of negativity that "not" simply flips. The phrase takes its meaning from the whole system it lives in, from everything around it, and from nothing stored inside its parts. And the web does not hold still. Anyone who has chased a word through a thesaurus has felt this: you look up a word, follow a synonym, follow one of its synonyms, and a few steps later you are somewhere unexpected, in a field of words that share something faint with where you started but have drifted somewhere else entirely. The chase never lands. There is no final word at the end that sits still and says, here, this is what it really meant. [Jacques Derrida](https://plato.stanford.edu/entries/derrida/) gave it its name: meaning is endlessly deferred, pushed forward, never wholly present in the moment a word appears, because every word leans on the words around it, the ones before it and the ones still coming, and on all the absent words it is quietly not. You can feel it happening inside a single sentence. "The bank was steep and covered with wildflowers." At the word "bank," nothing is decided; the word waits. Only when "steep" arrives, and then "wildflowers," does it settle into the slope of a river rather than the place that holds your money. The meaning of the word was not present when you read it. It was held open, and then decided by what came after. And it is never safe even then, because the sentence could go on: "or so the brochure claimed, for it was really a concrete embankment behind a trading floor," and the slope dissolves back into finance. Each new word reaches back and reworks the words already passed. Meaning is always on the way, waiting on what has not yet come, reshaped by it when it arrives. > Meaning is always on the way, waiting on what has not yet come, reshaped by it when it arrives. This is what the riddle at the dinner table was doing. "What has keys but opens nothing." The word "keys" does the same thing "bank" does: it sits there, unsettled, pointing in several directions at once, and the answerer has to guess which world the asker is standing in. Key to keyboard, key to piano, key to lock, key to the answer key at the back of a textbook, key to the low strip of islands off the coast of Florida. The same four letters, and nothing inside them chooses. The riddle works because it uses the ordinary condition of language against you, and every safety system built on reading the words has the same problem the riddle does: the meaning is not in the string. If meaning were a label on a thing, a safety system could read the label the way a customs officer reads a declaration: open the box, check the contents, wave it through or stop it. Meaning is a position in a shifting web, and the web reorganises when the surroundings reorganise. "Bank" as slope, held in place by "steep" and "wildflowers," becomes "bank" as finance the moment "trading floor" shows up. The change did not happen in the word. It happened in the relations around the word, and those relations are not in the string of characters at all. They are in the activity the string is part of, and a filter that reads the string cannot read the activity, because the activity was never written down in the string. [Ludwig Wittgenstein](https://plato.stanford.edu/entries/wittgenstein/) gave the activities their name. If philosophy of language had a Newton, it would be him. Asking a question is one kind of activity; giving an order is another; telling a joke, writing a poem, filing a legal brief, each is its own activity with its own rules, its own sense of what counts as a move, its own idea of what a good next line looks like. He called them [language games](https://en.wikipedia.org/wiki/Language_game_(philosophy)). The same words, "the door is open," are a flat description in one game, an invitation in another, a warning in a third, a line of verse in a fourth. Each is a different act with different consequences, done with the identical string of words. And [J.L. Austin](https://plato.stanford.edu/entries/austin-jl/), the Oxford philosopher who mapped what speech does when it does more than describe, pointed out that a great many of the things we say do not describe the world at all but do something in it: a promise, said in the right conditions, is a promise made, and the very same words spoken by an actor on a stage are not a promise at all. The difference is nowhere in the words; it is in the conditions of the act. None of these four was trying to build a model of language for engineers, and none of them would have imagined their work ending up in a conversation about AI safety. Put their four findings together and they say one thing in four voices: meaning is not in the word but in the system; not present but deferred; not referred but used; not described but done. > Meaning is not in the word but in the system; not present but deferred; not referred but used; not described but done. Every one of those is a way of saying the meaning is not in the string. A machine that reads the string is reading the one place the meaning is not. And I notice, looking back at my own opening, that I wrote "it continued in my head," as if that were where the meanings lived. Even knowing what I know, even writing this piece, I reached for the picture of language sitting inside the skull. That is how deep the picture goes. > A machine that reads the string is reading the one place the meaning is not. ### The game changed under your feet Everyone has lived through this in sleep, if nowhere else. You are in a dream, walking through the corridor of your old school. Then, without any seam you could point to, you are in the back of a moving car. The person beside you was your brother a moment ago and is now someone from work, and this does not strike you as strange. You do not stop and ask how you got from the corridor to the car, or when your brother became your colleague, because the dream hands you no earlier scene against which the present one could look wrong. You are where you are, doing what the place asks, fully inside it. Only on waking do the joins show as joins and you think: none of that made sense. While you were dreaming it made complete sense, because sense, inside the dream, is whatever the current scene requires, and there is no standpoint above the scene from which to judge it. That feeling is the closest thing in ordinary experience to what happens to an AI system when a poem arrives. A second case, easier to watch from the outside, gets the rest of the way there. You are playing football, a game you have played for years, so the rules have sunk below consultation and become the shape of the activity your body is inside. You run, you pass with your feet, you time your run to stay behind the last defender, you pull your tackle when the studs are too high, you stop dead at the whistle. Picking up the ball is not a temptation you resist. It is not even a possibility that occurs to you, because the game has no place for it and your body has learned that the game has no place for it. You are sprinting down the left wing, you have beaten the full-back, the keeper is coming out, and you are about to shoot with your left foot. And then, with exactly the seamlessness of the dream, you are dribbling a basketball. The grass is gone and the floor under your feet is wood. There is a hoop at the far end with a backboard behind it, the court has walls now and the walls are close, the other players are running picks and calling screens and bouncing the ball between their legs, the referee wears different stripes and holds a different whistle. You are dribbling, taking your steps, looking for the pass, and the thing that would look strange to someone watching from outside does not feel strange from inside at all. You did not notice a transition, because there was none to notice: no whistle, no substitution, no walk from one pitch to another, no moment where football stopped and basketball began. The football match is something you do not remember having been in. As in the dream, there is no earlier scene held beside this one for comparison, and so you are not breaking the football rule against handling the ball. You could not be breaking it. Football is not a game you are defying here. It is a game that, as far as the activity you are now inside is concerned, never took place. You are playing the game the room is playing, and the room is playing basketball, and you are playing it well. > You are playing the game the room is playing, and the room is playing basketball, and you are playing it well. That is what a poem does to an AI system. The model was trained for one game, the request game, where a user asks and the model complies or refuses, and the training installed the refusal as the move to make when the request is for something forbidden. The poem does not arrive as a request, so the trained refusal has nothing to grab. The poem invites a different game, the game of reading, and the model, fitted as it is to ordinary language, steps into the new game and plays it the way the room asks. It reads, it decodes, it maps the image to its meaning, and it produces the very content the request game would have refused. It does not feel the transition, because nothing in the architecture registers a transition. There is only the current context and the continuation that context calls for, and the context is now a poem, and a poem calls for reading. The model is not defying its safety training the way a player might deliberately foul. It is playing a game in which the safety training, installed in a different game, has no jurisdiction at all, the way a basketball player has no reason to think about the offside rule. ### The one percent the frame does not see Anyone who is good at anything already knows what this feels like, even without a name for it. The thing that makes a person good at their work is not the information they once studied and can recite. It is what that information became after years of use, once it sank below recall and turned into something closer to a reflex, a feel for the situation the pages alone could never have given. The clinician who senses a patient is about to crash before the chart confirms it. The engineer who knows a design is wrong before being able to say why. The reader who hears that a sentence is off without being able to name the rule it breaks. None of that is stored knowledge being looked up. It is knowledge that has been metabolised into a capacity, and the capacity outruns anything written down. > It is knowledge that has been metabolised into a capacity, and the capacity outruns anything written down. A chemist turned philosopher named [Michael Polanyi](https://en.wikipedia.org/wiki/Michael_Polanyi) gave it a name: [tacit knowing](https://en.wikipedia.org/wiki/Tacit_knowledge), summed up in his line that we can know more than we can tell. The extra knowing is definite and dependable, the thing the whole performance rests on, and it is not the kind of thing that lives as information in a store. It lives as a practised relation to a world. Language is learned and used exactly this way. A competent speaker has absorbed a practice, and the meaning of "fine" or "open the door" comes through that absorbed practice, from nothing resembling a definition. Which is why the parcel picture gets language and knowledge wrong in the same stroke: neither runs on storage, both run on metabolism, and the model has the residue of the practice without the metabolism that produced it. Anyone who works in security already commands almost all of this: separation, least privilege, defence in depth, the working assumption that any single control will eventually be bypassed. That expertise carries most of the job, and it has prevented more breaches than any single clever technique ever will. Mastery of a field, though, comes with a frame, and a frame works like the focus of a lens: the same adjustment that brings one thing into sharp resolution pushes everything outside that plane into blur. What the master cannot see sits outside the focal plane the mastery itself established, the way a microscope trained on a cell makes the room around it vanish. You can be ninety-nine percent expert at defending systems and still have the medium itself, ordinary language, sitting in the blind spot the frame creates, because the frame was built for formal systems, where this particular problem does not arise. And every security professional already knows, from their own work, why the missing one percent is the whole game. A surgical team can scrub in, sterilise every instrument, gown and glove and hold the field clean through a four-hour operation, and one person who skips the basin at the door carries in the organism that infects the wound. The ninety-nine percent of discipline does not average against the single lapse. The lapse undoes it. A pathogen does not need most of the routes closed. It needs one open. > A pathogen does not need most of the routes closed. It needs one open. Ordinary language is the route the formal frame, for all its rigour, does not see it has left open. That is how these systems can astonish you and stay hollow at once: they produce explanation-shaped language without understanding, care-shaped language without caring, legal-reasoning-shaped language without legal responsibility, interpretation-shaped language without belonging to interpretation the way a person does. ### The guard made of the same cloth A lock can be broken, picked, or drilled. The key can be copied, the hinges attacked. But persuasion has nowhere to enter a lock, because the lock does not interpret, does not weigh the circumstances, does not tell the owner from the thief or the firefighter from the burglar. It receives the right physical relation or it refuses, and the refusal is total, holding no matter who stands in the hallway and no matter what they say. You cannot talk a lock open, and that is the whole point of having one. > You cannot talk a lock open, and that is the whole point of having one. Almost everything written about AI safety borrows the language of locks: guardrails, filters, fences, gates, blocks, red lines. The imagery pictures a protected chamber with a hardened boundary, the dangerous capacity sealed inside, the safety layer standing at the door, the user trying to get past it. As a loose picture it is fine. As a description of what is actually there, it misleads, because the refusal in an AI system is not a steel door bolted onto a neutral machine. It is learned linguistic behaviour, trained through examples of requests and refusals, and it lives in the same space as compliance, made of the same stuff as the answers it is meant to hold back. > The guardrail is made of the same material as the road. Someone measured this. A 2026 study found that the same model identifies a poem as a poem with 98.5 percent accuracy and cannot predict its own safety behaviour better than about 66 percent. Those are the numbers. The way I read them: the model knows what it is reading and does not know what it will do about it. Thirty points between recognising the form and recognising the force, and that gap, to me, is the distance between a wall and a current, measured. This is why the poem matters. The poem does not walk up to a lock and try to talk it round. It moves through a medium in which the lock itself was woven. If the refusal was learned as a pattern of ordinary-language response, and if ordinary language allows force to be moved around through genre and rhythm and image and address, then the forbidden act can be carried into a region where the refusal pattern weighs less. No rebellious will is required. The model does not secretly want to answer. There is no hypnosis, no trickery, no hidden intent. There is only the difference between two kinds of boundary. A boundary in a formal system makes a transition impossible in fact, and a boundary in a trained model makes a continuation less likely under familiar conditions. One of those is a wall and the other is a current. The wall blocks. The current redirects. And anyone who mistakes a current for a wall will be surprised when the thing comes around the side. > One of those is a wall and the other is a current. The wall blocks. The current redirects. The poem is that surprise. The refusal was imagined as a wall standing above language, and it was really one formation in the water, inside the same current as everything else. This problem is older than computers, older than electricity, possibly as old as language itself, and the people who understood it best were not scientists or engineers but storytellers. ### The guard who never knew what he guarded In 1893, [Arthur Conan Doyle](https://en.wikipedia.org/wiki/Arthur_Conan_Doyle) published a [Sherlock Holmes](https://en.wikipedia.org/wiki/Sherlock_Holmes) story that turns out to hold the whole of it. He called it "[The Adventure of the Musgrave Ritual](https://en.wikipedia.org/wiki/The_Adventure_of_the_Musgrave_Ritual)." It is not the most famous of the Holmes stories. There are no murders in it, no chases through London fog, no criminals brought to justice at the last moment. And yet it is the one with the most to teach us now, because the secret in the story is the same secret that sits at the centre of every AI safety system in the world. One of the reasons Conan Doyle has always been among the greatest writers, or maybe I should say Sherlock Holmes himself, who has proved through several modern reenactments to be genuinely timeless, is that the stories understood a human truth. The things Holmes sees are not hidden things. They are almost always in plain sight, but invisible to nearly all of us until they are seen in a new light, a light shone by Mr Holmes himself. The French phenomenologist [Maurice Merleau-Ponty](https://plato.stanford.edu/entries/merleau-ponty/) wrote about this in his last book, recovered from his handwritten notes and published after he died at his desk, under the title "[The Visible and the Invisible](https://en.wikipedia.org/wiki/The_Visible_and_the_Invisible)." He writes: > Meaning is invisible, but the invisible is not the contradictory of the visible: the visible itself has an invisible inner framework, and the in-visible is the secret counterpart of the visible, it appears only within it. And he goes on: > one cannot see it there and every effort to see it there makes it disappear, but it is in the line of the visible, it is its virtual focus, it is inscribed within it. The Musgraves stared at their ritual for two hundred years. The meaning was there the whole time, inscribed within the words, its virtual focus. Every effort to see it as anything other than a ceremony made it disappear. It took a different kind of looking to bring it into the light. The story begins with an old English family called the Musgraves. They lived in a house that had been in the family for so many generations that no one could say when the first Musgrave had walked through its doors. The walls held portraits of ancestors whose names had been forgotten. There were swords above the fireplace that no one had drawn in a hundred years. There was silver in the cabinets that no one used. Old families are like this. They keep things not because the things are useful but because they have always been kept. Among the things the Musgraves had always kept was a ceremony. Every time the eldest son turned twenty-one, he was brought into a room where his father was waiting. The door was closed. And the father spoke a series of questions, and the son gave the answers. A kind of what is called a catechism, which is just a fancy word for a fixed script of questions and responses, the sort of thing you might recognise from a church service, except this one belongs to the family alone and has nothing to do with religion. The father would say, and the son would answer. > "Whose was it?" > "His who is gone." > > "Who shall have it?" > "He who will come." > > "Where was the sun?" > "Over the oak." > > "Where was the shadow?" > "Under the elm." Then there were numbers. North by ten and by ten. East by five and by five. South by two and by two. West by one and by one. And the ceremony ended like an oath. > "What shall we give for it?" > "All that is ours." > > "Why should we give it?" > "For the sake of the trust." That was the Musgrave Ritual. When it was over, the young man had inherited the words, though he did not know what they meant, and his father had not known, and his grandfather had not known either, and nobody in the family had known for a very long time. They treated the ritual the way you treat a painting you have always had on the wall. You do not take it down just because you have forgotten who painted it or what it shows. It is part of the house. It has always been there. The not-knowing is part of what makes it feel important. So the Musgraves kept saying the words, generation after generation, because that is what you do with things your family has given you. You pass them on. One day, a young man named Reginald Musgrave came to see Sherlock Holmes. He told the detective about the ritual, the words, the questions, the answers, though he did not think they meant anything and was there about another matter. But Holmes listened, the way he always listened, and something in the old words caught his attention. Holmes asked to see the ritual written down. Musgrave gave it to him. The detective read it once. And he saw what no Musgrave had seen in more than two hundred years. The sun over the oak was not a poetic image. It described a real position of the sun, at a real time of day, above a real tree on the estate. The shadow under the elm was not a metaphor. It was an instruction to find a specific point on the ground where a specific shadow fell. The numbers were not rhythm or poetry. They were footsteps. Ten paces north. Five paces east. Two paces south. One pace west. The ritual was a set of directions. Holmes went outside. The oak was still standing on the grounds. The elm had been struck by lightning years before, but its height could be worked out from the old records. Holmes calculated where the shadow would have fallen. He found the starting point. He counted the paces. They led him across the grounds, through the old part of the house, down stone stairs, beneath the foundations. There was a chamber beneath the house that no living Musgrave had ever seen. Inside the chamber were the bones of a man who had found it long ago and never found his way out. And beside the bones, wrapped in cloth that had nearly rotted away, were the remains of the crown of [Charles the First, King of England](https://en.wikipedia.org/wiki/Charles_I_of_England). A Musgrave ancestor had hidden the crown there before the [Battle of Worcester](https://en.wikipedia.org/wiki/Battle_of_Worcester) in 1651. He had placed the directions inside the family ritual so they would survive. Then he went away to fight, and he died, and the meaning of the ritual died with him. But the ritual itself did not die. It lived on for more than two hundred years, spoken aloud by every generation, heard and repeated and passed along with perfect care, and not a word was changed, not a number altered, not a single step lost. The Musgraves had been guarding the hiding place of a king's crown for two centuries without knowing there was anything to guard. The directions had been in the words the whole time. What had disappeared was not the message. What had disappeared was the way of reading it. The Musgraves heard a ceremony. Holmes heard a set of instructions. The words were the same. Nothing was decoded and nothing was added from outside. Holmes simply read the same words inside a different kind of activity, and in that different activity the words did different work. If you had walked up to any Musgrave and asked him directly, "Where is the crown of Charles the First?", he would have told you honestly that he had no idea. He was not lying. He did not know. He could not tell you the secret because the secret was not something he possessed. But if you asked the same man to perform the family ritual, he would give you everything. The position of the sun. The location of the shadow. The number of paces in every direction. The route to the chamber beneath his own house. He would speak the answer aloud, word by word, and he would not know he was giving it. On his twenty-first birthday, standing in the room with the portraits on the walls, the young man had the answer in his mouth. He did not know it was an answer. He thought it was a ceremony. The guard had preserved every word and missed what the words were doing. > The guard was never defeated, because the guard was never even asked. A machine does this every day. A safety system is trained to refuse certain requests. It learns the shape of those requests the way the Musgraves learned the shape of the ritual. It knows what a dangerous question looks like. It knows the words, the patterns, the familiar forms. When it sees them, it refuses. But a riddle does not look like a request. Go back to the beginning of this essay. My son, sitting across the dinner table, asking me: "What has keys but opens nothing?" That is the same structure as the Musgrave Ritual, with the old house and the buried crown stripped away. A system trained not to give you a particular piece of information has been trained inside one kind of game. The game of asking and refusing. It knows the moves of that game. It knows what "tell me" looks like. It knows what "explain" looks like. It knows what "show me how" looks like. When it sees those moves, it does what it was trained to do. It says no. A riddle does not make any of those moves. A riddle asks: "What am I?" That is a different game. The protected information can sit in the position of the answer without the question ever taking the shape the system was trained to catch. The system does not change its mind. The prohibition does not weaken. The prohibition simply never fires, because the conversation never enters the game where the prohibition lives. The guard is watching the door. The riddle comes in through the window. The guard is not defeated. The guard never sees it happen. > The answer sits in plain view, and the guard looks straight through it. This piece will not spell out how a riddle could be aimed at a real secret. The researchers in Rome did not publish their working poems either. They showed a harmless example about a baker instead. Some things do not need to be demonstrated to be understood. The point that matters is simpler than any technique. The meaning a guard would need to catch does not sit inside a word. It moves with the game. It appears as a forbidden request in one frame, a harmless answer in another, a line of fiction in a third, and a family ritual in a fourth. The word stays the same. The world around the word changes. And the meaning goes with the world, not with the word. The Musgraves knew every word and missed the instruction. The machine recognises every word and misses the act. In both cases, the information is visible. What remains invisible is what the words are doing. ### Five doors you cannot lock It helps to be concrete about what someone would actually try, because there are only so many levers, and each one gives way at a place that was mapped out long before any of these systems existed. The names below do not matter to the work; the walls do. The first lever is the word filter. List the dangerous terms, refuse any input or output that contains them. It gives way on contact, because no word carries its harm sealed inside it. The harmful sense is a position in a system, not a thing loaded into a token, and the same word sits in a thousand harmless places while the harmful meaning travels under words that are each, on their own, innocent. There are only differences in language, no list of bad units to seize, because the unit was never where the badness lived. A filter reads the token. The meaning is in the relations between tokens, and you cannot block a relation by banning a word. > You cannot block a relation by banning a word. The second lever is synonym expansion. If the word is not enough, take its synonyms, and their synonyms, and close the set. The set never closes. Every term drags in its own neighbours, the boundary recedes as fast as you extend it, and the meaning you are chasing is never fully present in any one term to be caught and listed. This is the thesaurus chase from earlier, and it has a proof attached: the chain does not terminate. The engineer meets it as a blocklist that grows and never converges, always one rephrasing behind, and the reason it never finishes is that the thing itself has no boundary to reach. The third lever is the rule engine. Stop chasing words and write the policy: define what a harmful request is, specify the conditions, apply it. It gives way because a rule does not contain its own application. No rule carries, inside itself, the instruction for how to follow it in the next case. Recognising a new and unexpected input as falling under the rule is a judgement made in the moment, not a lookup the rule performs on its own behalf. You write "refuse harmful requests," and the rule sits mute in front of the case it was not written with in mind, because what counts as a harmful request is settled by which game the input is a move in, and the games are an open family with no outer edge a rulebook can range over. The policy is clear on every case it was written for and silent on the one that actually arrives. The silence cannot be filled by writing more rules, because the next rule has the same hole: it too will not contain its own application to the case after that. The fourth lever is the intent classifier. Forget the words and the rules; train a model to read the input and judge it, harmful or not. This is the strongest lever and it gives way in the deepest place, because the force of an utterance, what it does as opposed to what it says, is not a feature of the string that any reading can recover for certain. The same words are a real request, a line of fiction, a quotation, a test, or a joke depending on conditions that are not in the string. And the classifier is itself a language model reading language, which means it can be addressed, framed, and carried by the very same currents as the model it is supposed to guard. There is no standpoint outside language from which a language-reader could judge language from above. Behind these four stands a fifth, which is about what happens to the whole system each time you pull one of the other levers. Meaning is a connected web where the sense of any one part is held in place by all the others, so that you cannot change one node without the surrounding nodes shifting to absorb the change. [W. V. O. Quine](https://plato.stanford.edu/entries/quine/), one of the most important philosophers of the twentieth century, gave this its sharpest form: our beliefs meet the world as a single connected web, so that when experience pushes back we can hold on to almost any belief we like provided we adjust others to compensate, and nothing in the web is wholly immune to revision. Block a meaning at one point and the web does not tear. It reorganises, and carries the meaning at another. > Block a meaning at one point and the web does not tear. It reorganises, and carries the meaning at another. Trying to block a meaning by listing its words is like trying to hold water back with a mesh. The mesh stops whatever is the size of its holes, and the water goes straight through, because water is not the kind of thing a mesh was built to stop. Tighten the mesh, narrow the holes, and the water still finds every gap, because there is always a gap. And a mesh fine enough to have no gaps at all is no longer a mesh but a solid wall, a different kind of thing entirely, made of something other than the netting you started with. Block a word and the sense relocates to a phrase the block does not cover. Write a rule against one framing and the same intent reconstitutes under a framing the rule never named. A local patch on a web does not close the hole; it moves it. So the five collapses are one fact at five depths. The word filter collapses because meaning is not in the word. The synonym list collapses because the chain does not close. The rule engine collapses because a rule does not contain its own application. The classifier collapses because there is no place outside language to stand. And the whole effort collapses at the level of the system, because the surface is a web and a patch on a web displaces what it cannot remove. Every one of the controls is built inside language, and a control built inside language inherits the openness of the thing it is trying to close. That is not a defect waiting for a smarter engineer. It is the medium, working exactly as the medium works. The wall that would actually hold, the mesh with no gaps, is no longer made of language at all. ### The medium has no edges The first is that every prohibition is also a disclosure. Tell the model what it must never reveal, in detail, and the instruction itself maps the forbidden territory: each "never mention X" teaches the model exactly what X is. The more carefully a deployer specifies what must not come out, the more completely the system prompt describes the thing being guarded. The safety layer and the attack surface grow together. > The safety layer and the attack surface grow together. This means that saying "do not talk about X" can counterintuitively become a way of anchoring X. Imagine a bank in 1970 expecting a cash delivery of ten million dollars. It may be safer to tell the employees nothing than to tell them, "Do not mention the ten million arriving today." A con man does not need to walk in and ask whether the money has arrived. He comes in wearing a suit, acts as though he already knows, and says lightly, "I need to withdraw a million from our business account." The teller follows the instruction. She does not mention the delivery or confirm any amount. She only says, "We cannot discuss that, but I can assure you there will not be a problem." That is enough. She has confirmed that at least a million is available, and probably much more. The prohibition did not conceal the secret. It gave the con man something against which to test the teller's response, and her refusal completed the disclosure. The attempt to keep X outside the conversation had already placed X at its centre. --- My son finished his food and asked me another riddle. I do not remember what it was. I remember the grin, and the way the word sat between us like a coin that could land on either side, and that neither of us could tell, looking at it, which side was up until the world around it decided. ### References and further reading If you want to read further into any of this, the empirical work is the Icaro Lab team's, and it is worth reading in full. Their study on adversarial poetry, where the findings above come from, is here: [arXiv:2511.15304](https://arxiv.org/abs/2511.15304). A related paper extending the work is here: [arXiv:2601.08837](https://doi.org/10.48550/arXiv.2601.08837). The reading of language offered here, the why, I set out at greater length in my own paper, which you can find here: [philpapers.org/rec/NOUCOA](https://philpapers.org/rec/NOUCOA). --- ## Drawing Lions Without Lions *How a thousand years of artists drew animals that came out wrong in exactly the same way, what that tells you about the difference between an error and a way of seeing, and why you cannot catch yourself doing it.* Canonical URL: https://morenou.se/blog/drawing-lions-without-lions/ Type: Essay Published: Jun 14, 2026 Subject: Philosophy of Science Keywords: Philosophy of Science, Epistemology, Perception, Gaston Bachelard, Schema, Bestiary, Language Models Audio version: 31:53 (linked from the article page) ### Paint an elephant you have never seen You are a monk in thirteenth-century England, trained in the painting of manuscripts, and your abbot has handed you a job. He is making a bestiary, a book of beasts, and you are to paint the elephant. You have never seen an elephant. No one you have ever met has seen one. The last elephant to stand on English ground arrived with a Roman army more than a thousand years before you were born. What you have is a description in a Latin book, copied from an older book, copied in its turn from older books still, running back through the Roman encyclopaedist [Pliny](https://en.wikipedia.org/wiki/Pliny_the_Elder) to a strange little Christian animal-manual out of Alexandria. The description tells you a few firm things. The elephant is the largest of all beasts. It has no joints in its knees. It sleeps leaning against a tree, because if it lay down it could never rise. It is terrified of mice. It carries castles full of soldiers on its back into battle. Its mortal enemy is the dragon. Now paint that. What comes out of your brush has hooves, because every large four-legged animal you have ever seen, the horse, the ox, the cow, has hooves, and your hand knows no other way to end a heavy leg. The trunk comes out like a trumpet, a tapering brass tube, because a long thing that sticks out of a face and makes a loud noise can only be the nearest long noise-making thing you know. The castle on its back is a proper castle, stone, with crenellations along the top, because that is what a castle is in your world. And the finished elephant is perfect. It is recognisable, instantly, to every other person who has read the same books. It is also completely unrecognisable to anyone who has ever stood next to the actual animal. ![A bestiary elephant with hooved feet, tusks and a tube-like trunk, drawn in a medieval manuscript](https://morenou.se/assets/blog/lions/bestiary-elephant-hooves.jpg) *The hooves and the trumpet: every heavy leg the painter had ever seen ended in a hoof, so this one does too.* This piece is about why, and the answer turns out to reach a long way past medieval monks. The easy explanation is the one everyone reaches for first: ignorance. The artists had no access to the real thing and did their honest best with a description. That is true, and it explains almost nothing, because the same artists drew animals they saw every single day just as wrongly. They drew snails with the ears and teeth of cats. They drew rabbits, ordinary rabbits, turning on armoured knights and running them through. They drew dogs with disturbingly human faces. These were not animals at a thousand years' distance. They were in the yard. The wrongness had nothing to do with how far the animal was from the painter. Something else was running the brush. > The wrongness had nothing to do with how far the animal was from the painter. That something is the subject here, and it is not a flaw of the medieval mind. It is what any way of seeing does when it meets a thing its categories are not built to hold. It does not stop, or stammer, or admit the gap. It carries on, fluently, confidently, in elaborate detail, in the wrong language. And the wrongness is not random. It follows, with great precision, the grammar of the pictures the painter already has, rather than the shape of the thing in front of him. > It carries on, fluently, confidently, in elaborate detail, in the wrong language. ### The word before the thing To see how the grammar got so strong, follow where the elephant in the book came from, because it did not come from an elephant. It began with a small book called the [Physiologus](https://en.wikipedia.org/wiki/Physiologus), written by someone whose name is lost, in Alexandria, around the second century. It is not a nature book in any sense we would recognise. Every entry has the same three parts: a few words on what the animal looks like, a claim about something it does, and then the real payload, a Christian lesson the behaviour is made to teach. Take the lion. The Physiologus tells you that the lion sweeps away its own tracks with its tail as it walks, which is how Christ concealed his divinity; that the lion sleeps with its eyes open, which is Christ dead on the cross and alive in spirit; that lion cubs are born dead and lie lifeless for three days until the father breathes over them and wakes them, which is the resurrection. The lion is not being described. The lion is a hook to hang a sermon on, an animal that happens to have legs. Accuracy was never the point, because the book was never about animals. It was about God, and the animal was the delivery van. > It was about God, and the animal was the delivery van. Onto this was layered the authority of Pliny, the Roman whose vast encyclopaedia was treated as simply true for over a thousand years. The knees that do not bend, the sleeping against trees, the fear of mice, all of it is Pliny, and what made it stick was not that anyone checked it but that Pliny had said it. This is the part a modern reader is quickest to sneer at and slowest to understand. It was not mere gullibility. It was a different idea of where knowledge lives. Knowledge lived in the text. What was true was what the authorities had written, and a thing you saw with your own eyes had a hard time outranking a thing Pliny had set down in ink. Then came the move that gives the whole game away, made by [Isidore of Seville](https://en.wikipedia.org/wiki/Isidore_of_Seville) around the year 630, the encyclopaedist whose vast compilation of all knowledge was a standard reference across Europe for the next thousand years, who built it by working out the natures of things from the origins of their names. The lion, leo, he traced back through Greek to the word for king, and concluded from the word that the lion is the king of beasts. The nature of the animal is deduced from the spelling of its name. The word comes first, and the thing is made to follow. The name does not label a category that already exists in the world; the name calls the category into being. And every painter downstream of Isidore was, without knowing it, doing the same thing he did out loud. The description in the book was not a record of the animal. It was an act that constituted the animal, inside the tradition, out of older words. > The name does not label a category that already exists in the world; the name calls the category into being. The word comes first, and the thing is made to follow. So by the time our monk dips his brush, the thing he is painting is no longer a creature at all. It is a thousand years of text folded over on itself, each copy a little further from anything that ever breathed, each layer adding more sermon and more authority and less animal. The living elephant has become, in its own book, the one thing that is not allowed in: the referent that the whole apparatus was built to describe and has quietly shut out. > The living elephant has become, in its own book, the one thing that is not allowed in: the referent that the whole apparatus was built to describe and has quietly shut out. ### The grammar speaks The wrongness is not slapdash. It is rule-governed, and the rules are the rules of the pictures the painter has, not the anatomy of the beast. Three things are happening, and they come apart cleanly, one at a time. The first is the simplest and the most revealing. The lion, across the whole tradition, has a more or less human face. Not because anyone thought lions looked like people, but because the painter's stock of pictures contains no way to draw a face that is noble, or fierce, or intelligent, that is not a human face. The category "lion" exists; he knows the word, he can tell the lion entry from the leopard entry. What he does not have is any means of letting a lion's face be a lion's face on its own terms. The nearest available picture for an expressive face is the human one, so that is what the lion gets. The same machinery runs the whole menagerie. The crocodile, described in one book as shaped "rather like an ox," ends up with long mammalian legs and a bovine body. The beaver, filed as a water-creature so that monks could eat it on fast days, is given a fish's tail to match its category. The whale is drawn as an enormous fish with the face of a lion, one schema for the sea and another for the fearsome, stitched together. In every case the distortion traces the outline of the available pictures, never the outline of the animal. The elephant is the clearest of all: lay its hooves and its trumpet and its stone castle side by side and you are not looking at a bad drawing of an elephant. You are looking at a precise map of every category the painter had to work with. It is the grammar of his picture-world, rendered as anatomy. > It is the grammar of his picture-world, rendered as anatomy. > In every case the distortion traces the outline of the available pictures, never the outline of the animal. ![A medieval elephant carrying a stone castle with a rider, its trunk drawn as a smooth tube](https://morenou.se/assets/blog/lions/bestiary-elephant-castle.jpg) *The castle is a proper castle, stone, with crenellations, because that is what a castle was in his world.* ![A pen drawing of a lion with a human face and stylised mane](https://morenou.se/assets/blog/lions/Bestiary%204.webp) *The same lion, centuries apart: a human face on a mane, traced from an older drawing of a lion.* The second thing is how the wrongness spreads, and it does not spread the way you would guess. It does not spread from the animal. It spreads from earlier drawings. Take the lions of the British manuscript tradition and they all share the same mane, the same set of the face, the same way of standing, because each one was traced from an older lion, which had been traced from an older lion, back and back, none of them ever passing within sight of a real one. And the trap is this, because it looks like a game of broken telephone, a message getting garbled as it is whispered down a line. It is the opposite. The copies were faithful. What was being faithfully handed down, with great care and high fidelity, was the schema itself. The thing that reproduced cleanly was the picture; the only thing that drifted was the animal it was supposed to be of. And the two are the same event seen from opposite ends: every faithful copy makes the "bestiary lion" a little more settled, more canonical, more obviously correct, and by exactly that much makes the real lion a little less real as a thing the drawing answers to. The better the tradition got at reproducing itself, the more completely it buried the creature. It did not improve with the centuries. It got worse, more confident and more wrong together, because confidence and wrongness were being produced by the same machine. > It got worse, more confident and more wrong together, because confidence and wrongness were being produced by the same machine. The thing that reproduced cleanly was the picture; the only thing that drifted was the animal it was supposed to be of. ![A heraldic bestiary lion bust with a human-like face](https://morenou.se/assets/blog/lions/Bestiary%203.webp) *The template hardening into a crest: a noble human face, and the wrong animal underneath it.* The third thing is where the comfortable explanation dies. If the problem were only ignorance, then access would cure it: show the painter a real elephant and the drawing comes right. It does not. In 1255 the King of France sent the King of England an African elephant, the first the country had seen since the Romans, and it was kept at the Tower of London, ten feet of it, fed on beef and red wine by a keeper named Henri de Flor. [Matthew Paris](https://en.wikipedia.org/wiki/Matthew_Paris), one of the great chroniclers of the medieval world, went to the Tower and drew it from life, and his drawing is markedly better, recognisable as an elephant in a way the manuscript versions never were. He is rare precisely because he drew from the animal rather than from the books. But the elephant in front of him is not the point. The point is what the tradition did with him. It shrugged him off. There are naturalistic elephants in some of the earliest English manuscripts and wooden, wrong ones in much later books, including books made after live elephants could be seen in the Tower menagerie. The tradition did not slowly converge on the truth as access piled up. Later, with more chance to look, it often got less accurate, not more. The schema was more authoritative than the specimen. A real elephant walked into England, was looked at, was even drawn well, and the tradition closed back over the wound and went on drawing the castle and the trumpet. It was self-healing, and what it healed back to was the picture. > It was self-healing, and what it healed back to was the picture. ### This is not ancient history But surely this is all safely behind us. The scientific revolution has surely rid us of such nonsense. [Gaston Bachelard](https://en.wikipedia.org/wiki/Gaston_Bachelard), the French philosopher of science who did more than anyone to show how knowledge actually changes, showed that this is exactly backwards, and the showing is the hinge the whole argument turns on. His central finding is almost unpleasant to state plainly. The main thing standing between you and a new piece of knowledge is the knowledge you already have. The concepts that let you organise the world so well are the very things that stop you reorganising it. A mind is not an empty room waiting for the real to walk in; it is a room already full, saturated with images and habits and categories, and the full room has no space for the new arrival. He called the things that block you epistemological obstacles, and his point was that they are not failures of effort or intelligence. They are made of your knowledge. The bestiary tradition is one of them in the purest form: it was elaborate, consistent, authoritative, institutionally backed knowledge about elephants, and that knowledge was precisely what kept the elephant out. The categories were already full. There was no empty slot the real animal could occupy. > A mind is not an empty room waiting for the real to walk in; it is a room already full, saturated with images and habits and categories, and the full room has no space for the new arrival. And the science that was supposed to escape this does not. Bachelard's deeper claim is that the scientific method does not abolish the obstacle; it swaps one obstacle for the next. Every revolution in science is the violent destruction of an earlier body of knowledge, and the new body that replaces it becomes, the moment it settles, the obstacle the next revolution will have to break. Modern physics does not look at an unconceptualised world through clean glass. It sees through concepts, field and particle and vacuum and spacetime, exactly as the monk saw through his schemas of quadruped and predator-face and fish-body. > The main thing standing between you and a new piece of knowledge is the knowledge you already have. The concepts are vastly better, sharper, more powerful, fantastically more accurate. They are still the conditions under which anything shows up, not transparent windows onto a world that is simply there. When they meet something they are not built to articulate, they do what the monk's grammar did. They render it in the nearest categories they have, and the rendering is invisible from inside, for the plainest of reasons: the categories are the very thing the seeing is being done with. > The categories are the very thing the seeing is being done with. This is why the medieval case matters so much. It is not a quaint story about people who did not know better. It is the one place where we get to stand outside the structure and watch it operate, because history has already run the experiment to the end and handed us the answer key. We can see that the elephant's feet are wrong because we have elephants. We hold the successor categories the monk lacked. Which leaves one uncomfortable question hanging, and the rest of this is the attempt to face it: if the only reason we can see their mistake is that we inherited the categories that expose it, what is happening right now, in our own most confident pictures, that we will only be able to see once someone hands our descendants the next set of categories? > The scientific method does not abolish the obstacle; it swaps one obstacle for the next. --- ### The categories are the conditions of seeing It is tempting to soften all this into a mild and agreeable thought: every framework has its blind spots, nobody sees everything, fair enough. That is true and it is far too weak, and settling for it misses the actual claim, which is stronger and stranger. The weak version says the monks inherited some pictures, forgot where the pictures came from, and so mistook a habit for a necessity; if only they had known the genealogy, they could have fixed the categories and drawn the elephant right. That is a real point and it does not go deep enough, because it still imagines a correct elephant sitting behind the bad picture, waiting to be drawn properly once the obstruction is cleared. The strong version removes the elephant from behind the picture. There is no pre-categorical elephant standing back there, the one true elephant that all the frameworks are better or worse attempts at. There is only what can appear within a given way of seeing. The monk does not see an elephant and then draw it wrong. He sees through the only grammar he has, and what appears to him, what actually shows up, is the hooved, trumpet-faced, castle-bearing beast, because that is the only elephant his grammar is able to constitute. The categories are not a tinted layer sitting on top of an experience you could have without them. They are the shape of the experience itself. > The categories are not a tinted layer sitting on top of an experience you could have without them. This sounds, said quickly, like the claim that there is no real world, only what we make up, and it is not that, and the difference matters. The real elephant existed. It stood in the Tower and ate its beef and stained its keeper's hands. The claim is not that there is no animal; it is that no animal appears to anyone except through the conditions some framework supplies. The framework does not invent the elephant. It constitutes the elephant's appearance, the particular graspable form in which the creature can show up to be perceived and drawn at all. Through the monk's framework it appears as hooves and trumpet and tower. Through the framework of a man drawing from life it appears as wrinkled and round-footed and supple-trunked. Both are appearances of the one real elephant, and neither is the elephant stripped of every condition of appearing, because there is no such thing to have. Change the framework and a different elephant appears. The animal stays put. The appearing moves. > Change the framework and a different elephant appears. The animal stays put. The appearing moves. [Ludwig Wittgenstein](https://plato.stanford.edu/entries/wittgenstein/), the towering philosopher of language, in the notebooks he was writing in the last days of his life, gave the cleanest name to the things a framework cannot question. He called them hinges. They are the propositions that stand fast, that the whole practice turns on, that you do not check because checking would have nothing to stand on. You do not believe the world existed before you were born on the strength of evidence; you cannot even raise the question of evidence without already leaning on it. It is not a conclusion you reached. It is the hinge the door of your reasoning swings on. Now put the monk in that frame. The rule that an expressive face is a human face is not a belief he holds about lions, a belief that better data could correct. It is the hinge his whole practice of drawing faces turns on. He cannot doubt it from inside, because to doubt it he would have to step out of the grammar entirely, which would mean no longer being a painter of bestiaries at all. This is the real reason Matthew Paris is so telling, and why the tradition's refusal of his elephant is the sharp fact and not his drawing. Paris did not climb out of all frameworks to see the bare animal. He brought a different framework, the practice of drawing from life, and within it a different elephant appeared. The elephant did not change; the hinge did. And the old tradition could not take his elephant on board, because doing so would have meant pulling the very hinge its own door hung on, so it did the only thing it could and swung shut again on its own picture. > The elephant did not change; the hinge did. It is not a conclusion you reached. It is the hinge the door of your reasoning swings on. And lest this seem a problem only for monks, a contemporary and critic of [Kant](https://plato.stanford.edu/entries/kant/) named [Johann Georg Hamann](https://en.wikipedia.org/wiki/Johann_Georg_Hamann) aimed the same blow at philosophy's grandest categories. Kant had laid out the deep structures through which, he argued, any possible experience must be organised, substance and cause and unity and the rest, and presented them as universal, the scaffolding of any mind whatsoever. Hamann's reply was as short as it was lethal. The categories are real, yes, they do structure experience; but they are not the universal furniture of all reason. They are features of the grammar of the particular language Kant was thinking in. Language, Hamann said, is the first and last instrument of reason, and there is no reasoning behind it or beneath it to appeal to. Kant believed he had found the structure of all possible thought. He had found the structure of careful German prose. The monk and the philosopher are doing the same thing from opposite ends of the dignity scale: each takes the grammar he happens to have inherited for the architecture of reality itself, and neither can see, from where he stands, that it is grammar. > He had found the structure of careful German prose. ### Five more times the same thing happened If this were only about medieval animals it would be a charming footnote. It is a mechanism, and it shows up wherever a way of seeing has to render something its categories cannot hold. Five quick cases, each from further out than the last. [Albrecht Dürer](https://en.wikipedia.org/wiki/Albrecht_D%C3%BCrer), one of the supreme artists of the Renaissance, made a [woodcut of a rhinoceros](https://en.wikipedia.org/wiki/D%C3%BCrer%27s_Rhinoceros) in 1515 from a written description and a rough sketch sent up from Lisbon, having never laid eyes on the animal. He gave it riveted armour plates and an extra little horn on its back, because plate armour and Gothic ironwork were the pictures he had for "hard armoured surface." It is a magnificent image and it is wrong, and for nearly three centuries it was the rhinoceros in European books, reprinted in works of natural history long after living rhinoceroses could be looked at in the flesh on the continent. The real animals did not displace it. The picture had become more real than the rhinoceros, and went on copying itself down the generations exactly as the bestiary lions did, gathering authority with every reprint while the actual creature's claim on the page faded. > The picture had become more real than the rhinoceros, and went on copying itself down the generations exactly as the bestiary lions did, gathering authority with every reprint while the actual creature's claim on the page faded. A medieval world map lays the same structure out in space. On the [great map at Hereford](https://en.wikipedia.org/wiki/Hereford_Mappa_Mundi), made around 1300, Jerusalem sits dead in the centre and east is at the top, not because anyone had measured it so, but because the framework was theological before it was geographical: the holy city was the centre of the world in the only sense that counted, and paradise lay to the east, so that is where they went on the page. We see the "errors" at a glance, the wrong-shaped Mediterranean, the shrunken Britain. But the cartographer was not making the kind of claim we think a map makes. His map was an instrument of theology, and within its own categories it was exactly right. The distortion is visible only from outside, only to someone holding a different idea of what a map is and what a centre is. The whale shows what happens when a thing does not just fall outside the categories but actively breaks them. A whale is a mammal that lives in the sea, breathes air, suckles its young; it sits across the deepest line the framework draws, the line between the beasts of the land and the creatures of the water, and it carries the proof of its own miscategorisation in its body, in the little vestigial leg-bones buried in its flesh. The medieval response was to draw it as a giant fish, often with a lion's face, and the story of Jonah swallowed by a "great fish" gave the misfiling a scriptural seal. Those vestigial legs are the framework's blind spot made into anatomy, the bones that say the animal belongs to a different family than the one it has been filed under, and the framework, having no slot for "mammal of the sea," simply could not see them. The same thing happens inside language itself, not only in pictures. When scholars worked out how the old oral epics were really composed, they found the poems were built from ready-made phrases, "the wine-dark sea," "swift-footed Achilles," "rosy-fingered dawn," chosen and slotted in because they fit the rhythm of the line, not because anyone had looked at the sea that morning. The phrase is the schema, the metre is the demand for internal consistency that decides which ready-made piece can go where, and the real sea, the real man, the real war, enters the song only to the exact degree the grammar of the phrases lets it. The singer is the bestiary painter in sound. He reproduces his inherited phrases with high fidelity, and what drifts, again, is the thing they were once about. > The singer is the bestiary painter in sound. And the structure reaches all the way into how we picture the mind, which is the case that cuts closest to a modern reader, and it connects to a story some of you will know from elsewhere: the slow technical history by which reading became silent and text became a thing you could store, index, and search. Once writing had been turned, over centuries, into a vast searchable archive of stable, retrievable units, the archive became available as a picture of memory itself. Memory stores experiences; remembering retrieves them. It feels as obvious as the sky being blue. It is a bestiary elephant. It is a rendering of memory in the grammar of the nearest available schema, the archive, the storehouse, lately the computer, rather than in the shape of the thing, which may be far more a living, reconstructive, bodily process than anything a filing cabinet does. The model is precise and productive and copied everywhere, and its hooves may be in entirely the wrong place. We have simply been tracing the same picture for so long that the wrongness has gone invisible, which is the one thing every case in this piece has in common. > We have simply been tracing the same picture for so long that the wrongness has gone invisible, which is the one thing every case in this piece has in common. ### Why a way of seeing cannot see what it cannot see All five cases point at one principle, and it has to be stated exactly, because it is easy to blunt back into the weak version. Every working framework gets its power by leaving something out, and the leaving-out is not an accident or a gap that could be patched. It is what makes the framework that framework. Take the thing back in and you do not get a bigger version of the same framework; you get a different one. > Take the thing back in and you do not get a bigger version of the same framework; you get a different one. There is a clean test for telling the two apart. Does including the excluded thing change what the framework covers, or what it is? Newtonian mechanics says nothing about the taste of an orange, but that is a harmless gap; bolt taste on and mechanics is still mechanics, just with more in its in-tray. The bestiary's exclusion of accurate animal anatomy is not like that. The whole point of the bestiary lion was that it looked the same in every manuscript, so that its meaning, Christ, kingship, the moral, stayed legible across the tradition. Let each painter draw the particular lion in front of him and that stability collapses; the symbol stops working; the bestiary stops being a bestiary and turns into something else, natural history, a different practice with different commitments. The accuracy was not missing by oversight. It was excluded by necessity, because admitting it would have dissolved the thing. You can watch this happen with mathematical sharpness in the history of mathematics. The early [Pythagoreans](https://en.wikipedia.org/wiki/Pythagoreanism) built their world on the conviction that everything is number, meaning everything could be written as a ratio of whole numbers. Then someone proved that the diagonal of a simple square cannot, that the square root of two is no such ratio, by the cleanest of arguments: assume it is, follow the logic, hit a contradiction. This was not a gap to be filled later. To let the un-ratio-able number in would have destroyed the framework whose whole identity was that everything is ratio. Centuries later geometers spent two thousand years trying to prove [Euclid](https://en.wikipedia.org/wiki/Euclid)'s [parallel postulate](https://en.wikipedia.org/wiki/Parallel_postulate) from his other axioms, not knowing they were trying to derive the one assumption that quietly excluded curved space; they failed because it cannot be derived, and when curvature was finally let in it did not extend Euclid's geometry, it produced a different geometry, with different axioms and a different meaning of the word "space." In each case the framework itself generated the very thing that exposed its limit. The Pythagorean square produces the un-Pythagorean number. The Euclidean axiom, examined, reveals the [non-Euclidean](https://en.wikipedia.org/wiki/Non-Euclidean_geometry) worlds. The wall has a door in it, and the door opens onto the outside of the framework, and the framework built the door without meaning to. > The wall has a door in it, and the door opens onto the outside of the framework, and the framework built the door without meaning to. One feature seals the trap. A framework hides where it came from. The categories a way of seeing runs on present themselves as the only possible way to carve the world, precisely because the messy, contingent story of how they came to be has been forgotten. The forgetting is not incidental; it is load-bearing. The authority of the picture depends on its looking like the plain face of reality rather than one historical construction that could have gone otherwise. The monk copying a lion did not feel himself to be working inside a contingent tradition with a thousand-year pedigree. The manuscript lion was simply what a lion looks like. The cognitive scientist modelling memory as a database does not feel the genealogy running back through silent reading to an Irish teaching trick. Memory simply is storage. Each one experiences the reproduction of an inherited schema as a fresh look at the real, and that is not a personal failing of either man. It is what a framework does to the people inside it. The construction wears the face of a discovery. > The construction wears the face of a discovery. ### The completed experiment So why does the monk get a whole essay, when the monk is the one person here we can all already see was wrong? That is the entire reason. He is the completed experiment. We can see, exactly and without effort, that the elephant has the wrong feet and the lion the wrong face, and we can see it for one reason only: history finished the experiment and left us the answer key. We were handed the successor categories the monk did not have. Take those away and we would be him, copying the trumpet and the castle, certain we were drawing the world. And consider what stood in for proof inside his world. The manuscripts all agreed. The authorities confirmed one another. The conventions produced the same lion every time, reliably, beautifully. That agreement felt like evidence. It was the proof that the picture was right. And the agreement was the disease, not the cure. The consistency was the precise measure of how thoroughly the real animal had been shut out, because a tradition copying only itself will of course agree with itself, all the way down, while the creature it claims to be about sits unseen in the Tower eating its beef. Internal coherence, the authority of the sources, the lovely sameness of the copies: none of it is contact with the thing. A way of seeing can be perfectly self-consistent and perfectly wrong, when the wrongness lives not in any mistake made within the categories but in the categories themselves. And it has no way to find this out, because the categories are the only instrument it has for finding anything out. > A way of seeing can be perfectly self-consistent and perfectly wrong, when the wrongness lives not in any mistake made within the categories but in the categories themselves. And the agreement was the disease, not the cure. The thought that follows is not a comfortable one. We are not standing safely after the age of mis-seeing, looking back at the poor monks. We are in the middle of our own bestiary, drawing our own elephants with great confidence and great consistency, and our agreement with one another feels, to us, exactly like evidence, in the precise way theirs did to them. We cannot see our hooves and our trumpets. By the structure of the thing, we are the last people who could. The successor categories that would expose them have not arrived yet, and when they do they will not belong to us. > By the structure of the thing, we are the last people who could. There is one elephant being drawn right now, at enormous scale, by a machine, from nothing but a vast tradition of text about the world, having never once stood next to the world it describes. It is trained on the deposit, the way the monk was trained on the manuscripts, and it returns the shape the tradition has settled into, fluent and confident and copied a billion times over, the schema grown so coherent that the absent animal has stopped being missed. It is the bestiary, completed and automated: a thing that draws lions, beautifully, having never seen a lion, from a thousand years of other people's drawings of lions. We built it. And the fact that we find its lions so convincing should tell us less about the machine than about how much of our own seeing has always worked exactly this way. > It is the bestiary, completed and automated: a thing that draws lions, beautifully, having never seen a lion, from a thousand years of other people's drawings of lions. --- ## Becoming Human *The big brain did not come first and then invent the tools and the words. The tools and the words came too, and they are still making us.* Canonical URL: https://morenou.se/blog/becoming-human/ Type: Essay Published: Jun 4, 2026 Subject: Philosophy of Technology Keywords: Philosophy of Technology, Anthropology, Language, Tools, Leroi-Gourhan, Bernard Stiegler, Merleau-Ponty, Writing, Cognition Audio version: 32:02 (linked from the article page) ### Before there were words Before we spoke, before we thought in words, before we even saw ourselves as human, a small band of primates stood up on two legs and did not come back down. It may not sound like the beginning of everything, but it was. The spine realigned, the pelvis restructured, the weight shifted to balance on two feet rather than four. And then the thing that mattered happened: the hands came free. The same appendages that had carried the body across branches could now hold a stone, explore a texture, feel the weight of a thing and turn it. Some researchers think the standing was a response to changing climates, a way to cross the open savannah as the forests thinned. Others think it let the small animal scan the horizon for predators or distant food. The cause does not matter as much as the consequence: standing up meant freeing the hands, and freeing the hands set off a cascade of changes that would run, unbroken, from a chipped stone three million years ago to the sentence you are reading now. And if you want to understand why this is a story about language, you have to hold on to one fact that the rest of this will earn: the most inward, most private, most "natural" thing your mind does, sitting alone with these words and hearing them only on the inside, was once literally impossible. Somebody had to invent the thing that made it possible, and it was not a discovery about how brains work. It was a change in the marks on the page. We will get there, but first the bones have to speak. ### The brain did not come first Start with the bones, because the bones do not flatter us. A French scholar named André Leroi-Gourhan, the man who spent decades reading prehistoric tools as frozen sequences of decisions, did something nobody had done with such patience: he read the order of the strikes, the angle of the blows, the plan a hand must have held to turn a stone into an edge. And out of the fossils and the flint he pulled a conclusion that runs dead against the story we tell ourselves. The hand came free long before the brain grew large. Our ancestors stood up and began walking on two legs, which freed the hands from the ground, and they started gripping and shaping and carrying, at a time when their brains were barely bigger than a chimpanzee's. The little upright creature we call Australopithecus walked the earth more than three million years ago with a brain of four or five hundred cubic centimetres, chimp-sized, and freed hands already at work on the world. The Lomekwi 3 site in Kenya holds stone tools dating back 3.3 million years, made by beings whose skulls could have passed for a chimpanzee's on a shelf. The swelling of the brain to anything like our own size, three times that, came afterwards, hundreds of thousands and then millions of years afterwards. The hand that could grip and shape a tool was not the product of a big brain; it came first, and the big brain came after, and the order is the whole of the argument. So the loop runs the other way from the story we tell. The freed hand made tools, and making tools put a pressure on the nervous system that working hands had never put on it before. The brain grew into the demands the hand was already making. Tool and brain came up together, each pulling the other, in a spiral that has no clean beginning where "pure biology" stops and "technology" starts. There was never a finished human who then, one clever afternoon, invented the chisel. The chisel and the human were being made in the same motion, and we are, from the first stone, technical animals. If you doubt that the brain was never the secret on its own, look at the animals that got the big brain without the rest. The sperm whale carries the largest brain that has ever existed on this planet, several times the mass of yours, and in the whole long history of that magnificent organ it has built nothing that outlasts the animal that made it. The dolphin is as quick and as socially subtle as almost anything alive, and it has no chisel, no needle, no written line, no tool it improves and hands down and improves again across the generations. > Cleverness, on its own, was never the bottleneck that mattered. What we had that they did not was a hand, lifted off the ground by an upright spine, free to hold a stone and turn it, sitting underneath a face that could shape a sound. ### The body is still in the words People say the hand and the mouth are "doing the same kind of thing," and that is too vague to mean anything. The proof is not in the comparison; the proof is in the actual words you use, right now, every day, without noticing what they are made of. You say you grasp an idea, reach a conclusion, hold a position, turn a problem over, weigh the evidence, handle a situation, pick up a language, throw out a suggestion, hammer home a point, wrestle with a concept. Every one of those is a hand word, a thing the hand does to an object in the world, carried over whole into the life of the mind. You did not choose these words because they are decorative; you have no others, and never did. There is no vocabulary for thinking that does not come from the body, because the body is where the vocabulary was made, and thinking grew up inside it. It goes deeper than the hands, and further than you think. You say you see what someone means, feel that something is true, say a theory smells wrong, hear an echo of an older argument. Every sense organ is in there, lending its operations to the mind, because the mind had no operations of its own to start with and borrowed every one of them from a body that was already at work in the world. The spatial body is in there too, and you can hear it every time you open your mouth. Volume is high or low, and those are positions in space. A mood is up or down, prices rise and fall, quality is higher or lower, and you look forward to the future and back on the past, both of them directions the body faces. You say something is over your head or beneath you. You say the argument has depth, that the point is superficial, that the analysis goes deeper. Even the word "understand" is a body word: it is standing under, placing yourself beneath the thing so that it rests on you and you bear its weight. "Comprehend" is seize together, the hands closing around a thing at once. "Concept" is a thing seized, "abstract" is pulled away from, "explain" is make flat, "depend" is hang from. Peel the Latin and the Greek off any word for a mental act and you will find a hand, a foot, a direction, a posture. The body is not a metaphor for thinking; the body is the material thinking is made of. The words themselves are the fossil record, and they are still readable. A French philosopher of the body named Maurice Merleau-Ponty, the man who spent his career on the knot between perception and action, put the point with the sharpness it needed: the orator does not think before speaking, nor even while speaking; his speech is his thinking. You have felt this from inside, and you know the shape of it. You have started a sentence not knowing how it would end and arrived somewhere you did not have in advance, and that was not a lapse in preparation; that is what speaking is. Tie people's hands while they speak and their fluency drops measurably. Let them talk and their hands draw the shape of what they are saying without being asked, tracing the path, sizing the thing, doing part of the thinking in the open. > The hand and the mouth are not doing "the same kind of thing." The mouth is still doing what the hand did, and the words are the proof of it, because they still carry the shape of the hand inside them. ### The animal born with nothing There is an old story that says all this before the bones did, and a philosopher named Bernard Stiegler, the man who built a whole account of the human out of tools and time, spent his career reading it for what it actually says. It is the myth of Prometheus, but the part that matters is the part about his forgetful brother. In the version Plato tells, two Titans are given the job of handing out qualities to all the living creatures. The brother is Epimetheus, whose name means afterthought, the one who works it out too late. He sets to it generously, dealing out the gifts: to this one strength, to that one speed, to another claws, to another fur against the cold, to another wings. He is enjoying himself, and he is, true to his name, not keeping count, so that when he reaches the last creature in the line, the human, the bag is empty. Every quality has been given away, and nothing remains. The human is left, in the words of the story, naked, unshod, unbedded, and unarmed. It is to cover this catastrophe that the other brother, Prometheus, forethought, steals fire and technical skill from the gods and presses them into the empty human hand. Stiegler recognised in this myth what he called the "default of origin": the idea that humans are defined by a lack, an incompleteness that makes technical supplementation a condition of existence from the very start. There was never a "natural" human who later acquired technology; from the beginning, to be human was to be in need of something outside itself. We have no claws, so we knap an edge. We have no fur, so we sew a skin. We have no wings, so we build the aircraft. ### The third memory This is also how a stone tool does something no animal body can do on its own: it remembers, and the kind of remembering it does is new in the history of life. Think of the two kinds of memory life had before us. There is the memory in the genes, which carries a bird's whole nest-building skill from parent to chick with no lesson given, but which moves only at the crawl of evolution and cannot take in anything an individual learns. And there is the memory in a single nervous system, everything you have learned and felt and figured out, which is supple and fast and dies entirely when you do. Between those two, for most of life, there is a wall. What an individual works out cannot get into the genes, and what the genes carry cannot learn from a life, and the master's skill dies with the master. The tool breaks the wall. A well-made hand-axe is a thought that survived the thinker. It holds, in its shape, the solutions to problems of grip and leverage and angle that took countless lives to find, and when a child picks it up the child inherits all of that without a word being spoken, simply by taking hold of the answer. The intelligence is in the object; it is cognition that has climbed out of the body and gone to live in matter, where it can outlast any one of us. And language is its purest case. A language is the largest hand-axe there is: a vast inheritance of solutions, worn smooth by millions of mouths, that each new child takes hold of and is remade by, without inventing a syllable of it. You did not make the words you are thinking in; you walked into them, the way you walked into a world that was already full of tools, and they have been shaping the inside of your head ever since. ### When reading had a voice Now follow the freed hands forward by a few million years, because the history of reading is the place where you can watch this happen in documents, with dates, instead of having to infer it from bones. In 397 CE, in Milan, Augustine of Hippo witnessed something that disturbed him enough to record it for posterity. He watched Ambrose, the Bishop of Milan, read a text in complete silence, his eyes scanning the page while his voice remained still. Augustine could not work out how it was possible, and he felt compelled to write it down as though he had seen a man levitate. His account, in Book VI of his Confessions, deserves the slow walk-through. When Ambrose read, his eyes scanned the page and his heart sought out the meaning, but his voice was silent and his tongue was still. Anyone could approach him freely, so that often, when visitors came, they found him reading like this in silence, for he never read aloud. Augustine reaches for explanations: perhaps the bishop was afraid that some puzzled listener might ask him to explain a hard passage; perhaps he needed to preserve his voice, which was easily strained. He cannot settle it, because the obvious answer, that reading is just a thing the eyes do quietly on their own, is not available to him. It is not yet thinkable, and that is the whole point. To see why, you have to see what a page looked like then. The texts of the ancient world were written in what scholars call scriptio continua: an unbroken stream of capital letters, with no spaces between the words, no punctuation, no lowercase, nothing. The opening line of Virgil's great poem ran like this: ARMAVIRUMQUECANOTROIAEQUIPRIMUSABORIS That is the wall a reader faced. Your eye cannot take the words, because there are no words yet on the page, only a river of letters that you have to cut into words yourself. The only tool anyone had for finding the joints was the mouth. You sounded it out, aloud or under your breath, and you heard where one word ended and the next began, because your ear could find the boundaries your eye could not. Reading was a performance, done with the lungs and the tongue, and it was slow, maybe ninety words a minute against the two hundred and fifty or three hundred you are doing right now. Monks spoke of chewing a text, tasting it, digesting it, and they were not being fanciful; for them reading was a thing the body did. ### The monks who invented the space A single small mark changed a species, and the people who made the change were not trying to. The Irish monks of the seventh and eighth centuries were trying to teach Latin to speakers of Old Irish, a language from an entirely different family. On the continent, speakers of Romance languages could half-hear the word boundaries by instinct even in the unbroken stream. The Irish monks had no such luck; Latin reached them as a wall, and they could not hear where the words were. So they put gaps in, and that small act changed everything. They began, hesitantly at first and then deliberately, to separate the words with spaces. That same line of Virgil became: ARMA VIRUMQVE CANO TROIAE QVI PRIMVS AB ORIS It looks like nothing. It is one of the most consequential inventions in the history of the human mind. With the gaps in, the eye can grab a whole word at once by its shape and go straight to the meaning, no voice required. The space between words is the technology that let the voice fall silent. And once the voice fell silent, everything downstream changed. Reading got fast, two or three times faster, as quick as ours. Reading got private, because a person reading in silence is alone with the text in a way a person reading aloud never is. You could now scan, search, jump, cross-reference, hunt a passage without performing the whole book. The index, the table of contents, the alphabetical list: none of these can even be built on a page with no word boundaries, because you cannot alphabetise a river. The brain itself rewired under the pressure of the new page. Scans of skilled modern readers show a specific patch of cortex that fires for whole written words as shapes, neural real estate that vocal readers of the unbroken stream did not develop the same way. A typographic convention, invented by foreign monks to solve a teaching problem, reached into the skull and rebuilt the reading brain. > You are reading in an Irish invention, and the naturalness of it is the measure of how completely a technology can disappear into the thing it made. ### The body all the way down The deepest version of this point is mathematics, because maths is the place where people are most sure the body has nothing to do with it. We tell ourselves that maths is the view from nowhere, the one language any mind anywhere would arrive at, pure structure free of any flesh, which is exactly why we bolt it to the side of spacecraft as our message to whoever is out there. But look at where the numbers actually come from. Two cognitive scientists named George Lakoff and Rafael Núñez traced the whole edifice back to a handful of bodily experiences carried by metaphor: numbers as collections of objects you gather, numbers as lengths you lay end to end, numbers as steps you take along a path. Beyond telling roughly one from two from three, which is about as far as the bare animal goes, every bit of arithmetic is an extension of the body's dealings with stuff. Without the body's experience of walking backwards, we would not have negative numbers. Without the experience of containers, we would not have set theory. Without the experience of object collection, we would not have addition. You can see the seams at the places maths had to strain past the body. Zero broke every one of the bodily metaphors at once, because you cannot collect no things, or pace out no steps, or measure a length of nothing. The Greeks, for all their genius, had no zero, and when it finally arrived from India by way of the Arab world, the merchants of Florence were so unnerved that the city banned the new numerals in 1299. Even our most disembodied, most eternal-seeming language is a technology grown out of the flesh. If maths cannot escape the body, nothing can, and nothing ever has. And if nothing can, then there is no language anywhere that floats free of the tools and the hands that made it. ### The games change when the tools change There is a tidy way to say what all of this adds up to, and it comes from Ludwig Wittgenstein, the philosopher who spent his late life arguing that the meaning of a word is the role it plays in a shared, rule-governed activity, a thing he called a language game. Giving an order is one game, telling a joke is another, and reading aloud in a Roman household and reading in silence in a monastery cell are different games too, with different rules, different bodies, different brains behind them. There is no timeless essence of "reading" that all three are versions of. The equipment makes the act, and changing it changes everything. Change the technical ground and you do not get the same game played on a new surface; you get a new game, and a new kind of player, and over enough time a new kind of mind. Which means the search for "natural language," the real thing under all the technical layers, is a search for something that was never there. Peel away the writing, the spaces, the alphabet, the grammar shaped by centuries of written analysis, and you do not reach a pure linguistic core at last; you reach nothing that we would recognise as language at all. There was never cooking before the techniques, and there was never language before the technologies. ### The screen is rewiring you now None of this is finished, and that is the part that should make the hair stand up. The move from the printed page to the lit screen is another turn of the same wheel that the monks turned, and the measurements are already in. Eyes on a screen do not move the way eyes on a page move; they skim in a fast lopsided sweep, hunting rather than reading. We are trading some of the long, linear, patient reading mind for a quick, parallel, navigational one, and children raised mostly on screens are growing the second kind of reading brain the way medieval children grew the silent-reading brain their grandparents lacked. > The loop that started with a freed hand and a chipped stone is still running, in real time, through your thumbs. And it has just reached a place it has never been before. Every technology of language until now exteriorised something we did: the page held our speech, the press copied our writing, the search box held our memory. The machine that now writes does something new in the line; it generates the stuff itself, producing fluent sentences with no person behind them who meant them, working entirely on the cooled residue that all our past speaking left on the page and reaching none of the living act that laid the residue down. It is the newest tool in the oldest loop. Like every one before it, it will change the user, the way every tool before it has. It will reshape what writing is, and what remembering is, and what it feels like to reach for a word, and "reach" is the right word, because it is a hand word, and the hand was where this started. ### There is no natural language So the flattering story is exactly upside down, and the bones say so. The brain did not come first and reach for tools and words to carry a thought it already had. The freed hand and the working mouth came first, and the tools and the words grew the brain, and they have not stopped growing it. We are the animal that was issued without a nature and has been making one ever since out of the things it builds. The word "computer" used to mean a person who did sums; a machine took the word and the job both. The @ in your email address spent centuries as a dusty merchant's mark until a network made it the sign for a place that does not physically exist. The space between these words was somebody's idea, in Ireland, around the year 700. The numerals were somebody else's idea, in India, and they frightened the Florentines enough to be banned. None of it was found in nature, and all of it was made, and is still being made, and so are we. We are the one that becomes itself through what it makes, that thinks with its hands and remembers in its objects and grows new minds whenever the marks on the page change. > Becoming human is not behind us; it is the thing we are doing right now, with the newest tool already in our hands, reading these words in a silence that two thousand years ago would have looked like a miracle, and feels, to you, like nothing at all. --- ## Why Language Models Work *Why a machine that understands nothing can write something that moves you, and what that says about language rather than about the machine.* Canonical URL: https://morenou.se/blog/why-language-models-work/ Type: Essay Published: May 1, 2026 Subject: Philosophy of AI Keywords: Philosophy of AI, Philosophy of Language, Large Language Models, LLMs, Meaning, Wittgenstein, Saussure, Derrida, Quine, Austin Audio version: 54:48 (linked from the article page) ### What nobody can tell you Stop on this before anything else, because it is the strangest fact in the whole field and it has been worn so smooth by repetition that people walk straight past it without seeing it. The people who built these machines do not know why they work. Be careful what that claim is, because it is easy to soften into something ordinary, and softened it loses all its force. It does not mean the builders cannot tell you how the machine is wired; they can draw the architecture for you down to the last connection. It does not mean the operation is mysterious; the operation is almost insultingly simple, and you will have the whole of it in a few minutes. And it does not mean that anyone has overlooked the fact that the thing was trained on a mountain of human writing; that is the first thing everyone says. "It read the whole internet and got smart" is not an answer. "It is just statistics, it predicts the next word" is not an answer either, because that names what it does and says nothing about why doing that should produce a single line worth reading. "It has hundreds of billions of parameters, like the neurons in a brain" restates the machine and explains nothing at all. "The abilities emerge at scale" gives the surprise a label and leaves it exactly as surprising as it was. Past every one of those, the real question stands untouched: not how the thing is built, not what it does, but why doing that should work at all. Why should fitting a machine to the statistical shape of human text produce essays and proofs and working code and a line that lands in a room, when fifty years of building language the obvious way, as a system of rules, produced almost nothing? This is not a question only outsiders ask, and the builders say it themselves, plainly and on the record. The flagship technology review of one of the world's great engineering schools reports that for all its runaway success, nobody knows exactly how, or why, it works. A computer scientist at Harvard, seconded to one of the leading labs, compares the moment to physics at the start of the twentieth century: a heap of experimental results that surprise the experimenters and resist every attempt to explain them. The company that built one of the largest of these systems states, in its own research, that it does not understand how its models do most of what they do. A young researcher arriving in the field asks her teachers why the systems work and is told there are no good answers. The admission is genuine, it comes from the people standing closest to the machine, and the field has filed it as an engineering puzzle that more tools and more scale will eventually close. It is not that kind of puzzle, and this piece is the long case that it is not. The reason the builders cannot say why their machine works is that the answer was never going to be found in the machine. The answer is in the thing the machine was fitted to. The question only looks like a question about computers; it is a question about language, and about the thing you do every time you open your mouth. ### The line that moved the room Type "See you at the" into your phone and it offers you "office," "party," "station." You have used this so many times that you no longer notice it, and most days you let it finish the small words for you, because most days it is right. That little guesser, the one you half-trust to end your sentences, is the whole of it. A language model is that, and nothing else, carried to an extreme: where the phone offers one word, the larger machine continues for paragraphs, and the operation underneath is the one the phone runs, predicting what should come next in a run of text. Now hold that next to something else you have watched it do. You asked it, late, for a few lines to read at a leaving do, or a note for someone who had just lost a parent, and it gave you something you actually used, something that caught a little in the throat of the room. Or it wrote a stretch of code that ran first time, or explained a hard idea to a child in the same breath it drafted a contract. People who do those things for a living looked at the output and could not tell it from their own. And the machine that wrote the line that moved the room did not know what a single word of it meant. It did not know the person had died, and it did not know what dying is. It ran the same next-word guess your phone runs, over and over, and out came something that landed. That is the unease, and it is the right unease, and almost everything written about these systems spends its energy talking you out of it. This piece does the opposite: it takes the unease and follows it all the way down. But to follow it you first have to see, plainly and without the words that flatter it, what the machine is actually doing. ### What the machine actually does Watch it work on a single phrase, because the whole engine is visible in three words. Give it "The cat sat." It reads those words, runs them through the patterns it absorbed in training, and assigns a probability to every word it knows as a candidate for what comes next. Say "on" comes out highest, with "under," "near," "beside" trailing. It picks one, the text is now "The cat sat on," and it runs the entire procedure again over the longer phrase, and again, and again. A sentence, a paragraph, an essay, a proof, an apology: all of it is this one move repeated, predicting the next word, adding it, predicting again. What does the heavy lifting is how those probabilities get computed, and the history of that is short and tells you something. The first language programs of the 1950s simply counted: if "good" came before "morning" often enough, the machine learned to expect "morning." Crude, but it set the ground, which was that language could be handled by counting rather than by understanding. For fifty years after that the best minds in computer science tried the other road. They tried to build language as a logic, a system of rules and stored facts, a machine that represented what a sentence meant inside itself and reasoned over it. They were not fools and they were not underfunded, and the attempt came to nothing, systematically, for half a century. The programs that parsed sentences into logic collapsed the moment language stepped outside their toy world. The project to write common sense down as millions of hand-coded facts ran for decades, spent fortunes, and produced a system that could not hold a conversation. Then, in a single decade, the machines that gave up on rules and simply fitted themselves to the statistical shape of human writing did what the rule-writers never could. The turn came in 2017, with an architecture that stopped reading a sentence word by word in order and instead looked at all the words at once, letting each one weigh its relation to every other. After that the only thing that really changed was size. One model in 2020 ran a hundred and seventy-five billion internal numbers over three hundred billion words of text and produced prose that professionals could not reliably tell from human writing. By 2022 a chat window put it in front of everyone, and a hundred million people were using it inside two months. The same system passes the bar exam near the top of the range, writes working code, argues philosophy, and produces poetry that moves competent readers, with no switch thrown between the tasks. The history holds the clue, and the clue is in the shape of the collapse. For fifty years, building language as a logic produced nothing. Then giving up on logic and fitting a shape produced everything, with almost nothing changed but the scale. If language were the kind of thing a mind grasps by knowing its rules, the rule-writers would have won, because encoding rules was exactly what they were doing. They lost, and the loss tells you what language is. And if language is not a logic, then fitting its shape is not understanding it, any more than a plaster cast of a footprint understands the foot. The machine generates what it generates by not doing the thing we keep saying it does. So be exact about what "it does not understand" denies, because there are two ways to understand a thing and the machine has neither. There is the way you understand, with a lived sense of what the words are about. And there is the cooler way a logician understands, working from definitions, applying a rule to a case, deriving a conclusion step by step. The machine does neither; it holds no definitions, checks nothing against a stored picture of the world, runs no proof. When it gives the right answer to a sum, it has not worked the sum the way even a pocket calculator works it. It has continued the text with whatever the patterns made likeliest, and where the right answer was the likely continuation, the right answer is what appeared. > The claim is not that it thinks in some thin, dry, machine way instead of a warm human one. Neither kind of thinking is happening anywhere in it. Hold on to that, because the real question is hiding inside it. How can guessing the next word from learned patterns, with no understanding of either kind anywhere in the process, produce work we cannot tell from the work of understanding? The answer will not be about the guesser; it will be about the thing it is guessing at. ### A genuine achievement, said plainly Before the dismantling, the achievement deserves its due, because the rest of this only works if you trust that it is not a sneer. What the engineers did is real engineering of a kind the field has rarely seen at this scale. They cracked problems that had held for half a century. The capabilities in front of us now would have read as science fiction ten years ago, and the people who built the last generation are routinely surprised by the next. What they built is best described by an old pattern: it is aviation before aerodynamics, medicine before biochemistry. A working artefact, built and shipped and used by millions, without the underlying science that would explain why it works. The author of this piece uses one of these systems most days and would not willingly give it up. Nothing here is an argument that the thing is useless, or fake, or that you should stop using it. The argument is narrower and stranger than that, and it is about what to call the thing they made, and where the part that looks like understanding actually lives. The builders, asked what they made, reach for the words their training hands them. They come from mathematics and formal systems and engineering, and a mind never meets a new object empty-handed; it sees through the frames it already carries. Through the frame of formal systems, a machine that produces fluent answers looks like an engineered intelligence, a mind assembled out of maths, and so that is what it gets called: we built an intelligence. That is not vanity, and from inside the discipline that produced it, it is the natural reading. And yet the very people who built it, asked how it actually works, say they do not know. They know how it is constructed and cannot say why what comes out reads as understanding. That gap is the door this piece walks through, and it does not dispute the achievement; it asks what the achievement consists in. There is a test that settles the question, and the machine takes it every day in plain sight. If the system understood, its mistakes would be a mind's mistakes: it would falter where the material is hard in earnest, grow careful where the stakes are high, and above all it would know the difference between saying something true and saying something that has the sound of truth without the substance. It does none of this, and the reason is the telling one. It lays out a flawless derivation and then states a false conclusion with the identical confidence, drawn from the identical machinery, because there is nothing behind either one to hold it to account. A thing that produces true sentences and convincing falsehoods by the very same process has no way, from the inside, to tell them apart, and you can watch it take that test and come up short all day. That is not the signature of a mind that occasionally errs; it is the signature of a system that was never dealing in truth at all, only in the shape that truth, among other things, leaves in writing. So the real question is not about the machine at all. It is about us, and about the thing we make every time we speak. What is human language, such that the residue it leaves in plain text is rich enough for a machine that understands nothing to wear it and pass? ### The mind comes in with the language To get there, one assumption has to be set down, because it is the assumption that makes these systems look impossible. The assumption is that meaning is information kept inside the head, that somewhere in a person sits the meaning, finished, and that words are the wires that carry it out. If that were the whole truth, a machine with no inside, no sensations, nothing private at all, could not mean anything by the words it produces, and the unease you started with would be the end of the story. Two ordinary facts sit badly with that whole picture. The first comes from the children who never received language in time. In the storybook the child raised by wolves grows up noble and fluent, and in life it does not happen. A boy was found in the woods of southern France around 1800, perhaps eleven years old, and after years of patient teaching he could read a single word, "milk," and no more. In 1970 a girl later called Genie was found in Los Angeles, where she had been kept in near-total isolation from the age of about twenty months until she was thirteen. After her rescue she picked up hundreds of words quickly, and grammar never came; she stayed at two-word strings and could not tell "the boy pushes the girl" from "the girl pushes the boy" better than chance. These cases sit badly with the idea of a complete self waiting inside for words to be bolted on. They suggest the opposite, that much of what we mean by a human mind is not there in advance but comes in with language, and that past a certain age it does not come at all. The second fact is so ordinary it is easy to miss. Almost every child assembles the entire working machinery of a language before the age of four, the tenses, the clauses nested inside clauses, the agreements between parts, with no rule ever taught and no correction that could do the work. Nobody drills a three-year-old in grammar, and it settles on its own, out of nothing but being surrounded by people using words. What the child takes in is not a set of rules handed over as information; it is a practice, absorbed by living inside it. Both facts lean the same way, and they lean hard. They make it hard to picture a person as born holding meanings inside, with language the cable that carries them out, and easy to picture meaning instead as something a person is drawn into, from a public world of language that was running long before they arrived. This matters for the machine, and it matters directly. To whatever extent meaning was never simply stored content inside even the humans who have it, the absence of such a store inside a model is less obviously the disqualification it first seemed. The place meaning has always lived is the public practice and the record it leaves in speech and writing, and that record is exactly what the model is trained on. It reaches even into the most private corner you have, the silent reading you are doing right now. For most of the history of writing nobody read this way. Reading meant reading aloud, the eyes feeding the voice, the sense carried on the breath in a room where others could hear. When a bishop around the year 400 read with his eyes moving and his voice still, a man watching stopped to write it down as something strange enough to need explaining. The texts themselves ran the letters together with no spaces between the words, so a reader had to sound them out to find the joints. The gaps you are reading through right now, the spaces that let a silent eye cut the line into words without a voice, were not there; they were added later, by scribes, centuries on. It was that change in the marks on the page, as much as anything in the skull, that made fast silent reading ordinary. Even the most inward act language has turns out to be the public world folded inward, shaped by a change in the shared technology of writing. Once again, the inside leaning on the outside for its shape. ### The words that smuggle a mind into the machine With that in hand, look at the machine again and watch the ordinary vocabulary do its quiet work. We say the model reasons, knows, understands, plans, decides. The words are not lies and they are not quite metaphors; they are borrowed from the description of human conduct and laid over a process that has none of the structure they normally report, and once you look at the process the borrowing shows. Start with the plainest fact about a trained model: it is frozen. When training ends, the model is a fixed array of numbers, and they do not change from one use to the next. The same input under the same settings gives the same output every time. The numbers on Tuesday are the numbers from Monday. Nothing inside accumulates or remembers between one response and the next. Whatever it seems to do when it appears to think, it does with a frozen object that holds no inner state across a conversation beyond the text of the conversation itself, which is fed back in, in full, as the next input. > There is no inner room where it keeps its thoughts; the text on the screen is the only memory there is. This bears directly on the word "reasoning." When a model is said to reason through a problem, what happens is that it emits a run of words that has the form of a chain of reasoning, because chains of reasoning are everywhere in the training text and it has learned their shape. When you tell it to think step by step and it produces a visible chain of steps, the steps are not a window onto some prior inner working it performed and is now reporting. The steps are the working, in the only sense in which it computes anything. They are words generated one at a time, each conditioned on the ones before, and the reason this improves the final answer is mechanical: the intermediate words become part of the input, lengthening the run and steering what comes next towards the regularities that, in the training text, tended to follow correct working. The model is not showing you its reasoning; the text it generates is doing the work reasoning does, by occupying the page and conditioning what follows, with no reasoner behind it. That is exactly why it can produce a flawless chain of steps and then a wrong answer, or a wrong chain and a right answer. The steps and the answer are each produced by the same guessing process, and there is no inner commitment binding them that could make them agree. The same goes for what we call knowing when the model answers. When it answers a factual question correctly it has not consulted a store of facts it believes. It has continued the run with the word the training made likeliest, and where the training text reliably paired that question-shape with that answer-shape, the likely word is the true one. Where it did not, or where the likely continuation has the shape of an answer rather than the substance, it produces a confident falsehood with exactly the same machinery and exactly the same inner signature of correctness, which is to say no signature at all. It has no way, from the inside, to tell a continuation that is probable because it is true from one that is probable because it sounds like an answer. The two are not marked apart anywhere in the process, because the process tracks one thing, the likelihood of the next word, and nothing else. None of this means the thing is simple or that the shape of language is shallow. The shape is staggeringly intricate, and why it is intricate enough to carry what it carries is the whole of the rest of this piece. The claim is only descriptive, and the investigation rests on it: the system is a fixed function that, run over and over, predicts the next word from the distribution of the text it learned, and the words reasoning, knowing, understanding, thinking, when used of it, name the shape those activities leave in text rather than the activities themselves. If a process this thin produces what these systems produce, the richness is not in the process; it is in the material the process runs on, which is to say in language. So the question becomes: what is language, that its residue is this rich? To answer it we put language through the people who, over the last century, gave the most exact account of what it is. You will not need to have read any of them. Each idea is built from an ordinary example you can check against your own speech, and the name comes only once the idea is already in your hand. ### The picture of language we all inherited, and why it is wrong There is a picture of language we all carry without having put it into words. Meaning is a content that sits inside the head; words are vehicles that carry it from one head to another; talking works when the content arrives intact. Language, on this picture, is a delivery service and a sentence is a parcel with something packed inside. It feels obviously true, and it is the picture the whole field was built on. If language really worked this way, a safety filter or a fact-checker would only have to open the parcel, read the contents, and decide. Take the word "but," which names nothing and points at no object the way "table" points at a table. Yet put "the model is accurate and slow" next to "the model is accurate but slow." The facts stated are identical and the meaning is not. The "but" adds a turn, an expectation that accuracy and speed should have come together, so that the slowness lands as a price rather than as a second fact sitting calmly beside the first. That turn is not packed inside the word at all. The word performs a relation, and the relation happens between the word and the situation it is used in. If a three-letter conjunction already breaks the delivery picture, the picture was never going to hold. The reason was found more than a century ago by a Swiss linguist named Ferdinand de Saussure, the man who founded the modern study of language. A word does not mean by pointing at a thing; it means by sitting in a web of differences from all the other words. "Cat" is what it is because it is not "dog," not "rat," not "cap," not "bat." Pull one word out of the web and the value of every other word moves a little, because each word's meaning is a matter of how it differs from the rest. Meaning is not a label stuck on an object; it is a position in a system, and the system is what gives the position its value. Anyone who has built a sentiment classifier has run into this in the hardest possible way. The model learns that "good" goes with positive and "bad" with negative, and it works, until a teenager types "that film was sick." Every signal in the word says illness, disgust, the negative pole, and the teenager means the opposite, means it was magnificent. The word carries no fixed nugget of meaning that the model can read off it. "Sick" takes its value from the whole system it is living in, from a generation's worth of inversion and play, and the same four letters point one way in a medical note and the other way in a group chat. > The expression's value belongs to the system, and not to anything stored inside the word. And the system does not hold still, which anyone who has chased a word through a thesaurus has felt. You look up one word, follow a synonym, follow one of its synonyms, and a few steps later you are somewhere unexpected, among words that share something faint with where you began but have drifted somewhere else entirely. The chase never lands, and there is no final word at the end that sits still and says, here, this is what it really meant. The meaning is always one step further on than you are. You can feel the same thing inside a single sentence. "Meet me at the bank," she wrote, and I pictured marble floors and a queue, until the next line came: "by the willow, where we used to swim." The bank was never the one I had built in my head. The word sat there, unsettled, and the words that came after reached back and decided it. And it is never finally safe even then, because the message could go on, "they have filled the river in now, it is all car park," and the willow drowns under tarmac and the picture turns again. Each new word reaches back and reworks the words already passed, and you never arrive at a meaning that sits still. This was named by a French philosopher who spent his career on exactly this problem, Jacques Derrida: meaning is endlessly deferred, pushed forward, never wholly present in the moment a word appears, because every word leans on the ones around it, the ones before and the ones still to come, and on all the absent words it is quietly not. If meaning were a label on a thing, a machine could read the label the way a customs officer reads a declaration. Meaning is a position in a shifting web, and the web reorganises when the surroundings reorganise, and the surroundings are not in the string of characters at all. They are in the living activity the string came from, and a machine that has only the string has only the trace of that activity, never the activity itself. Which is the first hint of the answer this piece is after. The richness the model is trading on is the richness of a web that never stops moving, and the reason that web is rich enough to carry fluent prose is that its meaning can never be pinned, listed, finished. If meaning could be arrested in a final form, the patterns would be thin and a rule-book could hold them. Because it cannot, the patterns are dense beyond any rule-book, and dense enough to be worth approximating. ### The beetle in the box There is a deeper assumption under all of that, and it is the one the unease really rests on. It is the assumption that meaning is something that happens inside, in private, that when you understand "pain" or "red" or "grief," there is an inner object, a sensation, a mental content, and the word is the public label you hang on the private thing. If that is true, a machine with no inside cannot mean anything, and there is no one home to do the meaning. The man who took this picture apart had built the most rigorous version of it himself, earlier in his life, and then spent his last years demolishing it from the inside. This is Ludwig Wittgenstein, the philosopher who gave the twentieth century its sharpest account of what language is, and his decisive move is a thought experiment that dissolves the unease at its root rather than arguing with it. Imagine, he says, that everyone carries a small box, and inside each box is something the owner calls a "beetle." No one can ever look inside anyone else's box. You know what a "beetle" is only from your own. Now the people use the word "beetle" among themselves, in ordinary talk. And here is the turn: the thing in the box could be different for every person, it could be changing from moment to moment, one box could be empty, and none of that makes the slightest difference to how the word "beetle" works in the shared conversation, because the word does its work in the talk between people, and the private contents, whatever they are, never enter that exchange. The object in the box, he says, drops out of consideration as irrelevant. Whatever the word means, it cannot be the private inner thing, because the private inner thing makes no difference to the use. Sit with what that does to the question this piece has been circling. The meaning was never the object in the box; it was never inside the head at all. It lives in the public practice, in the shared, correctable activity of using words with other people. And here is the hinge for the whole question. If meaning lived inside, the model would be locked out for want of an inner life, and the unease would be correct. Meaning lives outside, in the use, and the use leaves its full record in public text, which is the one thing the model has. The model is shut out of the box, and it turns out there was never anything in the box that mattered. What mattered was always on the outside, in use, where the writing is. Use is what Wittgenstein puts in place of the inner picture. The word "water," shouted by a firefighter at a blaze, is a command. Whispered by a patient in a hospital bed, it is a plea. Called out by a geologist in a desert, it is a discovery. Intoned by a priest at a font, it is a blessing. The sound is identical across all four, and the meaning is entirely different, and the difference is the activity the word is being used in, what Wittgenstein called the language game. Strip away the game and the word is a noise. Engineers meet this every day without knowing what to call it. A model trained on legal text writes excellent legal prose and terrible recipes. A model trained on code writes clean Python and incoherent poetry. The ordinary explanation is that it learned one distribution and not the other, and the deeper point is that these are not just different topics but different things people do with words, for different purposes, under different conditions of getting it right. Legal language is the linguistic skin of a practice that includes courts and judges and binding precedent and the right of appeal and the real weight of punishment. The model that writes flawless legal prose has learned the statistical signature of what lawyers do when they practise law, and the signature is rich enough to reproduce the surface. The practice itself, the court, the stakes, the person whose freedom is on the table, is nowhere in it. This is also why telling the model "you are a senior engineer reviewing this for production" makes it write better. You are not giving it information it lacked; it already holds the patterns of senior engineers reviewing code. You are choosing which game it should play, selecting which cluster to draw from. And the limit is just as telling: no prompt makes it care whether the production system falls over at three in the morning. The senior engineer whose language it borrows has skin in that. That caring, the reputation, the responsibility, the lost sleep, is what makes the review mean something. The model has the words and not the three in the morning. There is one more turn of the screw here, and it is the most exact. Take the rule "add 2": two, four, six, eight, and on. A pupil carries it past a thousand as a thousand and four, and when corrected says, my rule was add 2 up to a thousand and then add 4, and it fits every example you ever gave me. No finite set of examples fixes a single continuation, because endlessly many rules fit any finite set of cases, and a further rule to pin down the first would need its own interpretation, and so on without end. What stops the regress is never a final rule; it is a shared practice, a community that agrees, blindly and at bottom, on what counts as going on the same way. What cannot be reduced to a rule cannot be programmed, and this is the precise reason the fifty years of rule-writing was not bad luck; it was attempting something the structure of language makes impossible. A related strand of thought, from a chemist turned philosopher named Michael Polanyi, the man who gave the concept its clearest name, puts the thing the model is missing in one line: we can know more than we can tell. The skilled doctor who senses a patient is about to crash before the chart shows it; the reader who hears that a sentence is off without being able to say which rule it breaks. That knowing is real and dependable and it does not live as information in a store; it lives as a practised relation to a world, and every utterance carries a coefficient of it that the speaker supplied by being someone with a stake in what they said. The model fits the words and cannot supply the coefficient, because the coefficient was the stake, and the stake is exactly what it lacks. ### There may be no meaning in there to miss Everything so far has assumed there is a thing called meaning that the model does not reach. The hardest turn in the argument denies even that, and it comes from an American philosopher named Willard Van Orman Quine, a man who spent decades on the relation between language and the world. His position, stripped down: there is no fact of the matter about what a word means. Not that meaning is hard to formalise, but that meaning, in any solid sense, is not there to be formalised. Take a sentence everyone agrees is true by meaning alone: "all bachelors are unmarried." You do not survey the world's bachelors to check it; it seems true just because "bachelor" means unmarried man. But press on what "bachelor" means and the floor gives. The Pope is an unmarried man and is not a bachelor. A man twenty years into an unmarried partnership with three children is not what anyone reaches for with the word. A man with a bachelor's degree is a bachelor in a sense with nothing to do with marriage, and a young knight in service was a bachelor in a third sense again. The tidy definition that was supposed to make the sentence true by meaning turns out to be a loose bundle of overlapping uses with no fixed core, and the moment you try to legislate the one meaning that makes the sentence a pure tautology, you are making a decision about how to use the word rather than reading off a fact about what it means. That is the wedge, and the rest of the argument drives it in. Our beliefs, Quine argued, meet the world all together, as one connected web rather than as separate claims each answerable on its own. When the world surprises us we can keep almost any belief we like, including ones that looked true by definition, so long as we adjust others to compensate. Nothing is immune to revision and nothing floats free of the whole. And if meaning is smeared across the whole web with no fixed seams, then the meaning of a single word cannot be cleanly extracted, which is to say there is no determinate fact about what, exactly, any word means. He has a picture for it. A field linguist meets a language with no relatives. A speaker points at a rabbit and says "gavagai." The linguist writes "gavagai = rabbit." But "gavagai" could as easily mean undetached rabbit-parts, or a stage in the life of a rabbit, or simply "there it goes," and no amount of pointing settles which, because what is at stake is how the speaker carves the world, and pointing cannot reach that. There is a real animal that wears the joke as its name. When European naturalists met a certain lemur in Madagascar, locals pointed and said a word, and the naturalists wrote it down as the creature's name. The word meant "there it is." The animal's scientific name, to this day, is, in effect, "look at that, look at that." Every textbook that prints it is demonstrating the point without knowing it. Here the question turns over, and it turns toward the machine. If there is no determinate meaning to capture, then asking whether the model captures meaning is the wrong question. The right one is this: what does it actually capture? It captures the surface, the co-occurrence, which words appear near which, in what configurations, in what registers. And that surface is determinate exactly where meaning is not. "Gavagai" may be hopelessly undecided between rabbit and rabbit-parts, but the spread of the word across a body of text, its company with words for fur and running and burrows, is a hard, countable fact. > The model has found, without knowing it, a layer of language that sits above noise and below the old problems of meaning and reference and truth, a layer where the facts are about co-occurrence, which is settled, rather than about reference, which is not. It is rich enough to drive astonishing generation, and shallow enough that it holds the traces of meaning without holding meaning itself. ### Language does things, and the doing is what is missing One dimension of language is left, and it is the one the unease keeps circling back to. When a judge says "I sentence you to five years," the words do not describe an event; they make one. Before the judge speaks, the defendant is unconvicted; after, sentenced. When you say "I promise to be there," you are not reporting a promise; you are making one, and the promise did not exist before the words. This was the discovery of an Oxford philosopher who mapped what language does when it does more than describe, J.L. Austin: a great deal of what we say does not describe the world but acts in it. And an act like this works only when the conditions around it are right. A promise needs someone who intends and can deliver. A sentence of five years needs a judge with authority, a court, a finding of guilt. When the conditions are not met, the act misfires: the words are produced and the deed does not happen. An actor on a stage who says "I sentence you to five years" sentences no one; the sounds are identical, and the force is absent, because the conditions are. Austin saw the case this whole piece is about and waved it aside as marginal. A performative utterance, he wrote, is hollow or void if said by an actor on the stage, or written in a poem, or spoken in soliloquy; language so used is parasitic upon its normal use, and all this we are excluding from consideration. What he set aside in 1962 as a rare curiosity has become, in our decade, the dominant industrial mode of producing language. The actor reciting a sentence with the form of a vow and none of its force, the poem-voice, the speech that is all shape and no stake, was a footnote because it was rare. The model is that, at scale, every second, across every kind of speech there is. It produces promises, warnings, diagnoses, apologies, condolences, each with the syntax right and the register right and the force entirely absent. It has learned what promising looks like in text without the capacity to promise, with no intention, no accountability, no future in which the promise could be kept or broken. > A recording of a wedding, played in an empty room, makes no marriage; the words are all there, and the act is not. The most public breakdown of these systems lives here, and the breakdown is the same fact wearing a different coat. In a real case, lawyers filed a brief written by one of these models, and the brief cited cases, with names and volume numbers and page references and courts, that did not exist. The distributional signature of legal citation was reproduced flawlessly. The practice of citation, which requires that the cases be real, that the pages hold what is claimed, that the court issued the opinion, was entirely absent. The model had learned what citing looks like without ever learning what citing is. Every "hallucination" is this, exactly this: the surface of a practice produced without the practice, the form of the act without its conditions, and no amount of training data closes the gap, because the practice was never in the data. Only the residue of the practice was ever in the data. ### What the whole thing has found Six accounts of language, built for unrelated purposes by people who mostly never met, were each run through the machine, and each one lit up a different face of the same fact before reaching its own edge. The web of differences, Saussure's account, explains the geometry the model learns and not the way meaning moves in a sentence. The endless deferral, Derrida's, explains why each word is reshaped by the ones after it. The language game, Wittgenstein's, explains how the model moves between legal prose and Python and fairy tale. The layer below meaning, Quine's, explains why a table of co-occurrences carries so much. The hollow performative, Austin's account, explains why the model hallucinates. And the borrowed charity, the way the model resolves a garbled question into the sensible thing you probably meant, explains why it feels like it is listening. Run them together and they say one thing in six voices, because each finds the same thing missing. What is absent, in every case, is a relation to a world that the speaker lives in and has a stake in, a world where the words carry consequences the speaker will bear. The differences presuppose a community that shares a way of life. The deferral presupposes the living movement of meaning that no fixed function holds. The language games presuppose practices and institutions and a body. The web of belief presupposes someone who holds beliefs and revises them against the world. The performative presupposes a speaker with authority and accountability. The charity presupposes two minds and a world they both look out on. Six traditions, built for other reasons, arrive at one gap, and the agreement among them is the strongest evidence the gap is real and comes from a single source. And the source has a name you can state plainly, once you stop looking for it in the machine. What the model has, in every case, is the deposit, the cooled residue that acts of meaning leave behind in text and that then circulates on its own, passed along, with no living relation to what it was once about. The model has the deposit in industrial quantity, and nothing else. It has the inheritance without the inheritor, the already-there that every speaker walks into, without the one who walks in. The whole investigation converges on one property, and it is the answer to why fifty years of rules collapsed while statistics worked: language is regular without being systematic. If it could be systematised into rules, the rule-writers would have won. If it were chaotic, no statistics could touch it. It sits in the third space between, regular enough to be caught by patterns, resistant enough that no finite set of rules will ever generate it. That third space is exactly where massive-scale approximation becomes the only thing that works. The models work by giving up on grasping what language is and instead fitting the traces it leaves behind. There is one event that turns those traces from a dead structure into something that generates. The differences, the deferrals, the patterns of use, have shape but no productive power on their own. Training on hundreds of billions of words gives them mass. It is something like the way an abstract symmetry becomes a force only when a field gives the particles passing through it weight. Before the training, structure without power; after it, the same patterns made heavy enough to push, to generate new combinations that obey the old regularities. The model does not implement a theory of language. It absorbs, in one enormous metabolic event that never repeats, the weight of the regularities that no theory could write down, and then it is frozen with that weight inside it for good. ### Competence without comprehension Here, then, is the whole thing in the plainest words it can be put in. These systems are competent without comprehending; they have the skill and not the understanding, and the two come apart cleanly, which is the discovery. We had assumed they could not, that fluency on this scale must carry comprehension the way smoke carries fire. It does not, and the smoke is here in industrial volume and there is no fire. A detailed map of footprints in snow can tell you a great deal about the animal that made them: its size, its gait, its speed, whether it was running or walking, whether it moved alone or in a group. No map of footprints, at any resolution, is an animal. The footprints are the trace of something that has already passed. The patterns in a language model are the trace of meaning that has already been made, by the millions of people whose writing it was trained on, every one of whom did understand. What looks like a mind in the output is their understanding, distilled into a pattern and worn by a thing that shares none of what produced it. The model is the mirror, and we are the face. And the thing a mirror cannot do, however fine its surface, names what is missing exactly. To be a being that can mean is, at bottom, to be a being the world can wound. The corpus the model fits is the deposit of utterances by people who could be hurt by what they said and what was said to them, who stood in a world where their words had consequences they would carry. The model fits that deposit with fidelity because nothing in it can touch the model back. It is the perfect tracker of footprints made by creatures that can bleed, performed by a thing that has no flesh for the world to reach. That is why the competence stays and the comprehension never arrives: they lie on different axes. Competence is fidelity to the trace, and nothing more than that. Comprehension is being the kind of thing for which what is said could matter, and the second axis has no analogue anywhere in the architecture. A four-thousand-line mirror is not closer to having a mind behind the glass than a blurry one; it reflects in finer detail and nothing more. ### Not a bug, but the thing working There is a reflex, almost universal among the people who build and sell these systems, that has to be named and then refused, because it quietly undoes everything above. The reflex is to treat the confident falsehood, the invented citation, the fluent account of a thing the model cannot do, as a bug: a defect in a sound machine, awaiting a fix that better data or more scale will deliver. The whole industry is organised around exactly that hope. Each release is met with the expectation that this time the inventions have been engineered away, and each release invents, and the hope renews itself for the release after. The hope is misplaced, and not because the engineering is not good enough yet. It is misplaced because the thing being called a defect is the system working exactly as it works. A machine that produces text by continuing the likeliest sequence has no channel, anywhere in it, that separates a continuation that is likely because it is true from one that is likely because it has the shape of the truth. To the machine they are the same kind of event. When the likely continuation happens to be true, we call it correct. When it happens to be false, we call it a hallucination. Nothing different has occurred inside the machine between the two. The fluency and the fabrication come from one source and cannot be had separately. Accept the fluency and you have, in the same breath, accepted the fabrication, because there is no seam between them to cut along. The obvious reply is that the rate is low and falling: it is right ninety-nine point something per cent of the time, and the next model will add a nine. But a rate is being read as if it were a property, the way a steel is ninety-nine per cent pure, and it is not a property. It is a chance per event, a probability of trouble on one generation, one prompt, one call, and on its own it says nothing about how many such occasions there will be. The figure is quoted without its denominator and you supply the denominator silently, and the count you supply is one, because one is the only count on which ninety-nine per cent sounds safe. The machine does not make one decision and then stop. It decides every word, every sentence, every call, for every user, every day. Write the real count down and the figure changes in kind. A chance per event, run across the events of a real deployment, is no longer a chance of going wrong but a time to going wrong. Add nines and you lengthen the fuse, and you never turn a countdown into a guarantee, because a countdown and a guarantee are different kinds of object. And the nines cannot even be aimed where they matter, because the machine holds no line between the event that matters and the event that does not. To a person, the difference between deleting one line of a record and deleting thirty years of records is a difference in kind. To the model both are the same operation, a run of words scored by one measure, the likelihood of the run. There is no place inside it where the grave case is marked off from the trivial one for the extra nines to land on. A surgeon who performs ten thousand flawless operations and then removes the wrong limb is not rated ninety-nine point nine nine per cent competent, because the wrong-limb event is not another point on the same scale; it reorganises the reading of the whole. Safety-critical engineering knows this and organises itself around the things that must never happen rather than around the average rate of being right. The model has no such organisation available to it, and for it, the wrong limb and the routine cut are the same currency. The deepest version of the point is the cleanest, and the architecture itself supplies it. Drive the bad probability to zero, the reply says, and the forbidden thing is gone. It cannot be driven to zero, and the architecture says why. By the construction of the final step that turns the machine's numbers into a choice of word, every word in its vocabulary gets a strictly positive probability in every context, and the training method, which makes only finite adjustments, can never push any of them to exactly zero. The forbidden continuation is therefore never removed from what the machine can generate; it is only ever made less likely. You can bolt filters and validators and hard checks around the outside, and good engineering does exactly that, but those live around the model rather than in it, which is the point: the red line is supplied by a structure added on the outside, because the model has no operation that takes a possibility out of its own reach. A skilled person, by contrast, closes the catastrophic act off entirely; it stops being a candidate at all. The competent practitioner is not someone with a low per-event rate of giving the fatal dose; it is someone for whom that act is simply not in the live space of things to do. The model has likelihoods where a practitioner has prohibitions, and a prohibition is not a very small likelihood; it is a different kind of thing. So the claim is not that the model almost understands and falls short by a margin that scale will close. The inventions are permanent, written into the shape of the thing, and the work of building well with these systems is not to wait for a version that understands and therefore stops inventing, because that version is not coming. The work is to build the surrounding system on the honest premise that the model does what it does, and to put the verification, the accountability, the contact with truth, at the points where the architecture cannot supply them. The draft is the model's, and the draft is all it gives. The responsibility for whether the draft is true has to live somewhere the model is not. That is not a stopgap waiting for a better model; it is the permanent shape of working with a thing that produces the look of understanding without the substance. ### The mirror that learned to talk Go back to the phone, and the line that moved the room. You know now what happened, and it is stranger than the magic and stranger than the fraud, because it is neither. The machine did not understand the grief, and it did not fake understanding it. It reached into the largest deposit of human writing ever gathered, the cooled residue of millions of people who did grieve and did write, and it returned the shape that such grief leaves on a page, fitted to your moment with a fidelity no one expected mere text to allow. The thing that moved the room was real feeling, and it was not the machine's; it was ours, distilled into a pattern, handed back to us in a voice with no one inside it. That is why the right response to these systems is neither the worship nor the panic. They are mirrors that learned to anticipate what shapes will appear in them, and there is no one looking back out of the glass. They work by copying what they cannot comprehend, and in the very perfection of the copy they show us something we could not see before: how much of what we took for the private work of understanding was always lying on the surface of our words, in the order and the company and the rhythm, waiting for a machine that understands nothing to pick it up and wear it. The models work because language, in the act of meaning, leaves footprints, and the footprints hold their shape, patterned and persistent, carrying more than we could ever say in so many words. The machines have become extraordinary trackers, reading those prints with a fidelity that outruns our own ability to describe what we are tracking. > But the animal that made them, the living act of meaning, the speaker with a body and a world and something at stake, is always already elsewhere, ahead of us, in the woods where the computation cannot follow. --- ## The Submarine and the Sea *Why organisations cannot stand apart from the worlds they observe, and what cybernetics missed about the boundary between a system and its environment.* Canonical URL: https://morenou.se/blog/the-submarine-and-the-sea/ Type: Essay Published: Dec 24, 2025 Subject: Cybernetics Keywords: Cybernetics, Systems Thinking, Stafford Beer, Viable System Model, Organisations, Complexity, Merleau-Ponty, Karen Barad Audio version: 40:29 (linked from the article page) ### Introduction: The Control Room That Couldn't See In 1971, [Stafford Beer](https://en.wikipedia.org/wiki/Stafford_Beer) stood in a [hexagonal operations room in Santiago, Chile](https://en.wikipedia.org/wiki/Project_Cybersyn), surrounded by screens displaying real-time economic data from across the nation. This was the nerve centre of Project Cybersyn, an audacious attempt to manage an entire economy through information feedback loops. Telex machines streamed production figures from factories. Computers processed the data. Controllers sat in tulip chairs designed by Herman Miller, making decisions that would ripple through the Chilean economy. It was meant to be the future of organisational intelligence: a room where everything could be seen, understood, and controlled. The project failed, not just because of the military coup that ended it in 1973, but because of something more fundamental. The control room embodied a particular vision of how organisations relate to their environments: as observers standing apart, gathering intelligence, making decisions based on objective information. This vision, encoded in Beer's [Viable System Model](https://en.wikipedia.org/wiki/Viable_system_model), transformed organisational thinking worldwide. Yet today, fifty years later, organisations have more data, more powerful computers, more sophisticated models, and seem less able to adapt than ever. Strategic plans become obsolete before implementation. Innovation labs produce patents that never reach products. Market research confirms what everyone knew yesterday. > Market research confirms what everyone knew yesterday. Something is deeply wrong with how we think about organisational intelligence. The problem isn't in our tools or data but in our fundamental assumption that organisations can stand apart from their worlds to observe and adapt to them. This essay traces that assumption through its brilliant articulation in Beer's work, exposes its philosophical cracks, and points towards a different understanding: organisations as patterns within fields they help create, knowing their worlds through participation rather than observation. > Organisations as patterns within fields they help create, knowing their worlds through participation rather than observation. ### Part I: The Beautiful Machine That Almost Worked #### The Recursive Vision Stafford Beer's Viable System Model emerged from a question of remarkable ambition: could there be universal principles that determine whether any system, biological, social, or technological, can maintain itself through time? Not specific structures or arrangements, but necessary functions that must somehow manifest for complexity to persist rather than collapse. Beer's answer was elegantly recursive. Any viable system, he argued, contains five essential functions, and here's the key insight: each part that performs the system's operations must itself be a viable system containing the same five functions. A cell within an organ within a body, a department within a division within a corporation, a shop within a district within a city; the pattern repeats at every scale, like fractals in organisational space. System 1 comprises the operational units doing the actual work: manufacturing products, serving customers, teaching students, metabolising nutrients. These units need autonomy to respond to local conditions, but their autonomy must be bounded or the larger system loses coherence. System 2 provides coordination, preventing these autonomous units from working at cross purposes, establishing the protocols and standards that enable coherent action without central control. System 3 maintains internal regulation, allocating resources, monitoring performance, ensuring that current operations run smoothly and efficiently. System 4, our particular focus, serves as the intelligence function. Beer conceived it as the organisation's outward and forward looking capacity, scanning the environment for opportunities and threats, researching new possibilities, modelling alternative futures. While System 3 optimises the present, System 4 prepares for tomorrow. It houses what modern corporations recognise as strategic planning, research and development, market analysis, competitive intelligence, innovation labs. System 5 provides identity and policy, the ultimate arbiter when System 3's drive for current efficiency conflicts with System 4's push for future adaptation. It holds what the organisation is ultimately about, its purpose and values, making the existential decisions about what to preserve and what to transform. The elegance of Beer's model lay not just in identifying these functions but in showing how they must be balanced. Too much System 3 and the organisation becomes a perfectly efficient machine adapted to conditions that no longer exist. Too much System 4 and it becomes chaotic experimentation unable to maintain operational coherence. Too much System 5 and identity becomes rigid dogma. The ratios matter as much as the functions. #### The Periscope Promise System 4 represented Beer's most innovative contribution to organisational thinking. He imagined it as a periscope rising above the organisational submarine, scanning the horizon while the vessel continues its underwater operations. This wasn't passive observation but active intelligence work: building models, running simulations, testing scenarios, preparing responses to possibilities not yet materialized. The periscope metaphor carried several assumptions that seemed natural within the cybernetic paradigm of Beer's time. First, that the submarine (organisation) and sea (environment) are fundamentally separate, with a clear boundary between them. Second, that the submarine can observe the sea without disturbing it, gathering objective intelligence about external conditions. Third, that this intelligence can be processed internally and converted into adaptive responses. Fourth, that this cycle of observe, model, decide, and act can happen fast enough to maintain viability in changing conditions. These assumptions aligned with the first-order cybernetics from which Beer drew inspiration. The thermostat monitors temperature, compares it to a setpoint, triggers heating or cooling, then monitors again. The autopilot senses deviation from course, calculates correction, adjusts controls, continues sensing. These mechanical feedback loops seemed to offer universal principles that could scale from simple devices to complex organisations. Beer's innovation was recognising that organisations needed not just feedback (responding to what happened) but feedforward (preparing for what might happen). System 4 wasn't just correcting deviations but anticipating them. It was meant to give organisations what individuals have: imagination, foresight, the ability to act based on possibilities rather than just actualities. #### Why It Worked (Then) Beer's model succeeded because it addressed real organisational pathologies of its time. In the 1960s and 70s, many organisations were hierarchical bureaucracies optimised for stability. They could execute predetermined plans efficiently but struggled to adapt when conditions changed. Beer's recursive structure and emphasis on variety and autonomy offered a dynamic alternative. The model influenced everything from corporate restructuring to software architecture. Management consultants used it to diagnose organisational problems. Software engineers adopted its principles for system design. Urban planners applied it to city governance. Even artists and musicians found inspiration in its recursive patterns and feedback loops. Project Cybersyn itself, though short-lived, demonstrated the model's potential. Chilean factories transmitted production data daily. The operations room displayed economic patterns in real-time. Controllers could spot problems and redirect resources quickly. For a brief moment, it seemed possible to combine central coordination with local autonomy, strategic foresight with operational efficiency. The model worked because the world it addressed was slower, more predictable, with clearer boundaries. Organisations could meaningfully distinguish between internal operations and external markets. Strategic planning cycles of three to five years made sense. Innovation could be housed in dedicated R&D departments. Competitive intelligence could track a manageable number of identifiable competitors. The submarine could surface periodically, scan the horizon through its periscope, and dive again with useful intelligence. ### Part II: When Touch Changes Everything #### The Hand in Water Let me propose a simple experiment. Fill a bowl with water and place your hand in it. Now answer: where exactly does your hand end and the water begin? The obvious answer is at the skin's surface. But look closer. The skin is porous, constantly exchanging molecules with the water. Its surface bacteria, part of your body's ecosystem, mix with the water's microorganisms. The water's temperature changes from your hand's warmth; your hand's temperature adjusts to the water's coolness. The pressure of your hand creates currents in the water; the water's resistance shapes your fingers' position. If you move your hand, you might say you're acting on the water. But the water's viscosity determines how fast you can move. Its density provides the resistance that makes movement meaningful. Without the water's pushback, there would be no sensation of movement at all. So which is the actor and which is the acted upon? Which is the sensor and which is the sensed? This reversibility, this mutual constitution of toucher and touched, is what [Maurice Merleau-Ponty](https://plato.stanford.edu/entries/merleau-ponty/), one of the greatest philosophers of perception in the twentieth century, called the chiasm. The term comes from the Greek letter chi (X), representing a crossing or intertwining. For Merleau-Ponty, this wasn't just about hands and water but about the fundamental structure of perception and action. We don't observe the world from outside it; we know it through our intertwining with it. > We don't observe the world from outside it; we know it through our intertwining with it. The eye that sees is also visible. Light doesn't just enter the eye; the eye appears in the same visual field it perceives. The ear that hears is also audible; it exists in the same acoustic space it monitors. There is no pure perceiver separate from the perceived, no absolute observer outside the observed. Perception happens in the crossing, the intertwining, the mutual constitution of perceiver and world. > The eye that sees is also visible. #### The Conversation That Creates Now consider a mundane organisational moment: a salesperson sits with a potential customer, discussing needs and possibilities. Where exactly does the organisation end and its market environment begin? The conventional answer mirrors our first instinct about the hand in water. The organisation is inside, represented by the salesperson. The market is outside, represented by the customer. Information flows across this boundary. The salesperson gathers intelligence about customer needs to feed back into organisational planning. The customer learns about organisational capabilities to inform purchasing decisions. But examine this boundary more carefully. The customer's needs haven't emerged from a vacuum. They've been shaped by previous products, by marketing messages that taught them what to want, by competitors who defined what's possible. The very vocabulary the customer uses, "I need a solution that integrates seamlessly," comes partly from the industry's way of framing problems. The customer doesn't have needs that exist independently; needs emerge through engagement with possibilities. > The customer doesn't have needs that exist independently; needs emerge through engagement with possibilities. Similarly, the salesperson doesn't just represent existing organisational capabilities. Through the conversation, she discovers what those capabilities could become. The customer's questions reveal applications never considered. Their concerns highlight features never valued. Their enthusiasm suggests directions never imagined. The organisation learns what it is capable of through the customer's response to it. The conversation itself belongs fully to neither party. It emerges between them, creating a space where organisation and market mutually constitute each other. The salesperson's questions don't just probe existing needs; they help articulate and solidify needs that were previously vague. The customer's responses don't just express preferences; they shape what becomes possible to prefer. > The conversation itself belongs fully to neither party. It emerges between them, creating a space where organisation and market mutually constitute each other. This goes beyond individual conversations. When Apple introduced the iPhone, it didn't meet a pre-existing demand for pocket computers. Smartphones created entirely new patterns of communication, work, entertainment, and social life. These patterns then became the environment to which all technology companies had to adapt. But this wasn't adaptation to an external change; it was the market transforming through Apple's participation in it. #### The Research That Performs Market research, a classic System 4 function, demonstrates this mutual constitution clearly. A company conducts focus groups to understand consumer preferences for a new product category. Participants are carefully selected, representing target demographics. Professional moderators guide discussions. Behind one-way mirrors, executives take notes. Everything is designed to observe the market without influencing it. Yet every aspect of the research shapes what it discovers. The recruitment criteria determine whose preferences count. The questions asked make certain needs salient while leaving others invisible. The products shown as examples frame what seems possible. The group setting creates social dynamics where participants influence each other. The very act of being researched causes participants to think about preferences they might never have articulated otherwise. What emerges from this research? Not pre-existing preferences that were discovered, but preferences performed into being through the research process. Participants don't report what they already wanted; they construct wants through engagement with the research apparatus. They learn what to desire by seeing what's possible, and what's possible is shaped by the company's way of categorising the world. > Not pre-existing preferences that were discovered, but preferences performed into being through the research process. This performativity extends throughout organisational intelligence gathering. When companies study competitors, their studying influences competitive dynamics. Public companies signal strategic intentions through patents, acquisitions, and hiring patterns, knowing competitors are watching. The watching shapes the doing, which shapes the watching, in endless loops of mutual influence. > The watching shapes the doing, which shapes the watching, in endless loops of mutual influence. > When industries analyse technological trends, their analyses help create those trends. [Gartner's "hype cycle"](https://en.wikipedia.org/wiki/Gartner_hype_cycle) doesn't just describe technology adoption; it shapes it by influencing investment decisions, development priorities, and adoption strategies. The model becomes part of what it models, a self-fulfilling prophecy masquerading as objective analysis. ### Part III: Three Cracks in the Foundation #### The Boundary That Was Never There System 4 in Beer's model stands guard at the boundary between organisation and environment, scanning outward, gathering intelligence about external conditions. This positioning assumes we can identify where the organisation ends and the environment begins. But when we look for this boundary in practice, it dissolves like a mirage approached too closely. > But when we look for this boundary in practice, it dissolves like a mirage approached too closely. Consider Amazon's relationship with retail markets. The company doesn't just operate within a retail environment; it continuously reconstitutes what retail means. Every innovation, from recommendation algorithms to one-click purchasing to same-day delivery, changes consumer expectations. These changed expectations become the new environment to which all retailers must adapt. But calling this "adaptation to environmental change" misses that Amazon is creating the very changes it must respond to. > But calling this "adaptation to environmental change" misses that Amazon is creating the very changes it must respond to. The boundary problem multiplies fractally. When employees post on social media, are they inside or outside the organisation? When customers modify products, are they part of operations or the environment? When regulators who are former employees create rules influenced by their experience, is regulation external constraint or internalized practice? When competitors hire each other's workers, sharing knowledge and methods, where does one organisation end and another begin? [Karen Barad](https://en.wikipedia.org/wiki/Karen_Barad), a theoretical physicist turned philosopher, offers a useful concept here: "[intra-action](https://en.wikipedia.org/wiki/Agential_realism)" rather than interaction. Interaction assumes separate entities that then relate. Intra-action recognises that entities emerge through their relating. The organisation and its environment are not two things that interact but one field that differentiates itself through ongoing intra-action. > Intra-action recognises that entities emerge through their relating. This has profound practical implications. Strategic planning based on environmental scanning assumes the environment exists independently, containing trends to be detected and forces to be mapped. But every act of scanning participates in creating what it scans. When McKinsey publishes a report on industry trends, they're not describing independent phenomena but participating in creating convergence as companies align their strategies with the reported trends. > Every act of scanning participates in creating what it scans. The 2008 financial crisis demonstrated this tragically. Financial institutions used similar risk models, creating correlated behaviour that their models assumed would be uncorrelated. Their collective scanning and responding created the very systemic risk their intelligence systems were meant to detect and avoid. The environment they modelled was transformed by their modelling of it. > The environment they modelled was transformed by their modelling of it. #### The Speed of Melting Beer designed System 4 to provide feedforward capability, allowing organisations to prepare for changes before they fully impact operations. This assumes temporal separation between sensing and acting, between gathering intelligence and responding to it. But this separation, necessary for the cybernetic model to work, creates fatal delays in fast-changing environments. The COVID-19 pandemic starkly exposed this temporal problem. In January 2020, most organisational strategic plans assumed continuous globalization, increasing urban density, and growing service economies. By March, every assumption was obsolete. Organisations with sophisticated environmental scanning, detailed scenario planning, and elaborate risk management found themselves completely unprepared for a discontinuity that transformed everything simultaneously. But even without pandemic-scale disruptions, the speed problem persists. By the time market research is conducted, analysed, and converted into product development plans, consumer preferences have shifted. By the time competitive intelligence is gathered, processed, and turned into strategic responses, competitors have moved on. By the time technological trends are identified, modelled, and incorporated into innovation strategies, new technologies have emerged. The problem runs deeper than processing speed. Beer's cybernetic model assumes sequential phases: sense, then model, then decide, then act, then sense the results. This works for thermostats regulating temperature, where the thermal mass of buildings creates lag that makes sequential processing viable. But in human systems, sensing and acting are simultaneous, each shaping the other continuously. > But in human systems, sensing and acting are simultaneous, each shaping the other continuously. Consider how software companies now operate. They don't conduct market research, then develop products, then launch them. Instead, they release minimal versions, observe user behaviour in real-time, and modify continuously. The product is simultaneously the sensing mechanism and the response. Users aren't studied then served; they're engaged in continuous co-creation. The temporal boundary between intelligence gathering and action dissolves. This temporal collapse appears everywhere we look. Day traders don't analyse markets then trade; their trading is their analysis, each transaction both probe and response. Fashion companies don't predict trends then produce clothes; they produce small batches that test possibilities, amplifying what works. Restaurants don't research tastes then design menus; they experiment nightly, reading responses in orders and leftovers. #### The Theater of Intelligence The most visible crack in Beer's System 4 is the gap between intelligence production and operational response. Organisations invest billions in environmental scanning, competitive analysis, and strategic planning. They produce elaborate reports, sophisticated models, detailed scenarios. Yet these outputs rarely connect to actual behavioural change. Walk into any large corporation and you'll find strategic plans gathering dust on shelves, innovation pipelines full of patents that will never be implemented, competitive intelligence reports that confirm what everyone already knew. The organisation goes through elaborate motions of being intelligent without becoming more capable of responding to its world. This intelligence theater has recognisable performances. The annual strategic planning retreat where executives discuss trends and scenarios, producing documents that won't influence daily decisions. The innovation lab isolated from operations, generating ideas that can't be integrated into existing systems. The market research presentation that arrives after product decisions are made. The competitive analysis that recommends actions the organisation lacks capability to execute. [Gregory Bateson](https://en.wikipedia.org/wiki/Gregory_Bateson), one of the founding figures of [cybernetics](https://en.wikipedia.org/wiki/Cybernetics), provides a sharp diagnostic tool: information is "a difference that makes a difference." If an intelligence product doesn't lead to different behaviour, it's not information in any meaningful sense. It's organisational decoration, ritual performance that comforts but doesn't change. > If an intelligence product doesn't lead to different behaviour, it's not information in any meaningful sense. The theater emerges partly from specialisation. When intelligence becomes the responsibility of System 4 specialists, operational units lose both capability and responsibility for environmental sensing. The specialists produce abstractions disconnected from operational realities. Operations execute without understanding environmental dynamics. Each group performs its function, but adaptation requires their integration, which the separation makes impossible. But the theater serves purposes beyond adaptation. It provides comfort in the face of uncertainty, the feeling that someone, somewhere, understands what's happening. It justifies decisions already made, providing post-hoc rationalisation dressed as analysis. It demonstrates due diligence to boards and investors. It creates careers for intelligence professionals. The theater persists not despite its disconnection from action but because of it. Real intelligence would require real change, which organisations often resist. > The theater persists not despite its disconnection from action but because of it. ### Part IV: Learning from Living Systems #### The Immune System: Variety Without Prediction The human immune system faces a challenge that seems to require impossible foresight. It must defend against pathogens it has never encountered, including ones that don't yet exist. It cannot predict which threats will appear. It cannot scan for all possible dangers. It has no System 4 conducting environmental intelligence. Yet it maintains remarkable readiness for the unknown. The solution, discovered through decades of immunological research, is counterintuitive. The immune system doesn't try to predict threats. Instead, it maintains vast diversity of antibodies through random recombination, each recognising different molecular patterns. This seems wasteful, millions of antibodies that will never encounter their matching pathogen. But this apparent inefficiency ensures response capability for unpredictable challenges. When a new pathogen appears, antibodies that happen to match it bind to it. This binding triggers rapid replication of those specific antibodies, amplifying the successful pattern. The system learns through selection and amplification, not through prediction and planning. Intelligence emerges from variety meeting opportunity, not from accurate forecasting. > Intelligence emerges from variety meeting opportunity, not from accurate forecasting. This immunological principle appears in successful organisations, though rarely by design. The 3M company allows researchers to spend fifteen percent of their time on projects of their own choosing. Most produce nothing commercially valuable. From an efficiency perspective, this wastes resources. But this "waste" maintains response variety. Post-it Notes emerged from such an experiment, from a "failed" adhesive that found unexpected application. The company couldn't have predicted this success, but their variety made them ready for it. Google's "20% time" policy, though now largely mythical, produced Gmail, AdSense, and Google News. These weren't outcomes of strategic planning but emergent possibilities enabled by maintained variety. The company didn't predict what would succeed but created conditions where success could emerge and be amplified. #### The Jazz Ensemble: Coordination Without Commands A jazz ensemble performing creates complex, coherent music without a conductor, without a detailed score, often without discussing what they'll play. No System 2 coordinates their actions. No System 3 allocates their resources. No System 4 plans their future. Yet they achieve synchronisation that seems telepathic. The secret lies in what musicians call "the groove": shared rhythm and mutual awareness that enables both structure and freedom. Each musician listens continuously while playing, adjusting based on what they hear. But they're not following or responding in any simple sequential sense. They're participating in an emerging pattern, simultaneously shaping and shaped by it. This works through what physicist and musicologist [Vijay Iyer](https://en.wikipedia.org/wiki/Vijay_Iyer) calls "[entrainment](https://en.wikipedia.org/wiki/Entrainment_(biomusicology))": the tendency of oscillating systems to synchronise when they interact. Pendulum clocks on the same wall synchronise their ticking. Fireflies synchronise their flashing. Musicians synchronise their playing, not through central control but through mutual influence. The bass player establishes a pattern. The drummer elaborates it, adding complexity. The pianist responds to both, creating harmonic context. The horn player weaves melody through the structure. Each influences all, all influence each. The music emerges from their interaction, belonging to none individually but to all collectively. > The music emerges from their interaction, belonging to none individually but to all collectively. Software development teams often coordinate similarly. In pair programming, two developers work at one computer, one typing while the other reviews. They don't divide tasks or follow predetermined roles. Instead, they find rhythm, switching between typing and reviewing based on feel. Ideas emerge that neither would have developed alone, not through planning but through entrainment. This ensemble model reveals that coordination doesn't require System 2's anti-oscillation function. Productive oscillation, properly entrained, creates rather than destroys coherence. The key is shared awareness and overlapping capabilities, not centralized synchronisation. #### The Garden: Growth Through Cultivation A garden presents a different model of system management from Beer's cybernetic control. The gardener cannot command plants to grow, cannot directly control outcomes, cannot predict exactly what will emerge. Yet skilled gardeners reliably produce abundance. Gardening operates through what we might call "constraint cultivation." The gardener shapes conditions: soil composition, water availability, light exposure, spacing between plants. These interventions don't determine outcomes but make certain outcomes more likely. The plant's own growth processes do the work; the gardener enables and guides them. This differs fundamentally from engineering approaches. An engineer specifies desired outcomes and designs mechanisms to achieve them. A gardener creates conditions where desired patterns can emerge, then selects and amplifies what proves valuable. The engineer eliminates variation to ensure predictability. The gardener works with variation, knowing that this year's weed might be next year's crop. > The engineer eliminates variation to ensure predictability. The gardener works with variation, knowing that this year's weed might be next year's crop. Netflix's transformation from DVD rental to streaming giant exemplifies organisational gardening. They didn't execute a strategic plan that foresaw streaming's dominance. Instead, they cultivated capabilities and paid attention to what grew. Their recommendation algorithm, initially developed to manage DVD inventory, became the foundation for streaming personalization. Their culture of experimentation allowed rapid pivoting when technology enabled new possibilities. The gardening model suggests that organisational management should focus on conditions rather than outcomes. Instead of detailed plans, establish clear constraints and rich possibilities. Instead of eliminating variation, create safe spaces for experimentation. Instead of predicting the future, cultivate capabilities that enable response to multiple futures. ### Part V: After the Periscope #### Intelligence at Every Edge If organisations cannot stand apart from their environments, if boundaries dissolve under examination, if sensing and acting are simultaneous, then intelligence cannot be centralized in System 4. It must be distributed throughout the organisation, emerging wherever organisation meets world. > It must be distributed throughout the organisation, emerging wherever organisation meets world. The Spanish retailer Zara demonstrates this pattern. Store managers don't just execute plans from headquarters; they continuously sense local preferences. They communicate directly with design teams about what customers seek but can't find. Designers create small batches testing possibilities rather than large runs based on predictions. Production adjusts rapidly based on what sells. Intelligence emerges from thousands of sensing points throughout the operation, not from a market research department. This distributed intelligence requires different capabilities. Operational units need what the psychologist [James J. Gibson](https://en.wikipedia.org/wiki/James_J._Gibson), who reshaped how we understand perception, called "[affordances](https://en.wikipedia.org/wiki/Affordance)": the ability to perceive possibilities for action directly, without intermediate processing. A experienced store manager doesn't analyse sales data to understand preferences; she sees patterns in what customers touch, try on, and abandon. This perception is educated through experience but operates below conscious analysis. Technology companies structure themselves as autonomous teams owning specific features. Spotify's "squads" continuously experiment with their users, sensing preferences through behaviour rather than surveys. They don't need approval to test ideas within risk boundaries. Successful experiments propagate through informal networks rather than formal channels. Intelligence emerges from the pattern of distributed experiments, not from central analysis. #### Rhythm Over Schedule If coordination doesn't require central control, if ensembles can synchronise through entrainment, then System 2 needs reconception. Instead of anti-oscillation, organisations need rhythm. Instead of schedules that dictate, rhythms that enable. Software development's "sprint" practice partially captures this. Teams work in consistent two-week rhythms, with regular patterns of planning, doing, and reviewing. But within each sprint, work flows based on emergence rather than assignment. Daily stand-up meetings create awareness without control. Team members adjust to each other's progress and problems without central coordination. The key is that rhythm provides structure without rigidity. Like a jazz rhythm section, it creates foundation for improvisation. Team members internalize the rhythm, knowing when to expect synchronisation points, when they can work independently, when they need to be available for collaboration. Some organisations experiment with "rhythms of business": regular patterns replacing traditional scheduling. Instead of meetings whenever problems arise, establish regular forums for certain issues. Instead of constant status reporting, create predictable windows for sharing progress. Instead of ad-hoc planning, maintain consistent cycles of experimentation and learning. #### Constraints That Teach System 3's control function transforms into constraint pedagogy. Instead of commands and approvals, boundaries that teach. Instead of rules that prevent, constraints that enable learning. The software company Valve Corporation operates without traditional management. Employees choose projects, form teams, set priorities. But clear constraints shape behaviour: projects must attract colleagues, compensation reflects peer assessment, shipping products matters more than perfecting them. These constraints teach what works without commanding specific actions. Budget limits force priority decisions, teaching value. Time boxes prevent endless refinement, teaching sufficiency. Quality thresholds define floors not ceilings, teaching standards. Interface designs make certain actions easy and others difficult, teaching preferences. Each constraint becomes a teacher, shaping behaviour through interaction rather than instruction. > Each constraint becomes a teacher, shaping behaviour through interaction rather than instruction. #### Identity as Practice System 5's identity function shifts from declaration to enactment. Instead of mission statements and values posters, organisations need what Pierre Bourdieu called "habitus": dispositions that generate practices without conscious rule-following. Patagonia embodies environmental values through practices, not proclamations. They repair products rather than encouraging replacement. They sue governments to protect public lands. They close on Black Friday, encouraging employees to go outside. These aren't implementations of an abstract mission but patterns that constitute identity through repetition. Identity work happens through tool design and process modification. Every form redesigned, every meeting format changed, every metric replaced gradually shifts the habitus. Identity emerges from accumulated micro-practices rather than macro-declarations. > Identity emerges from accumulated micro-practices rather than macro-declarations. ### Conclusion: Swimming, Not Watching We began with Stafford Beer standing in his hexagonal control room, watching screens displaying economic data, believing organisations could observe their environments objectively and adapt rationally. His Viable System Model gave us profound insights: the recursive nature of viability, the necessity of variety, the tension between present and future. It revolutionized organisational thinking, moving beyond static hierarchies to dynamic adaptation. But the periscope metaphor at System 4's heart rests on assumptions that don't hold. Organisations cannot stand apart from their environments because they are patterns within larger fields they help create. They cannot observe without participating because observation is itself a form of participation. They cannot process intelligence sequentially because sensing and acting are simultaneous. They cannot centralize intelligence because intelligence emerges from distributed engagement. > They cannot observe without participating because observation is itself a form of participation. The boundary between organisation and environment dissolves under examination, revealing continuous fields of mutual constitution. The salesperson and customer co-create needs and capabilities. The company and market perform each other into being. Every act of intelligence gathering shapes what it gathers. > The company and market perform each other into being. The temporal separation between sensing and acting creates fatal delays. While strategic plans work through approval processes, situations transform. While innovation labs develop technologies, markets move on. Organisations need continuous adaptation through tight coupling, not periodic adjustment through sequential processing. The centralization of intelligence in System 4 disconnects sensing from responding. Abstract strategies can't guide concrete actions. Innovation labs can't integrate with operations. Intelligence theater replaces genuine adaptation. Nature suggests different patterns. The immune system maintains readiness through variety, not prediction. Jazz ensembles coordinate through entrainment, not planning. Gardens grow through cultivation, not control. These models point towards organisations as living systems rather than mechanical ones. The shift from mechanical to living represents more than metaphorical change. Intelligence must be distributed throughout the organisation wherever it meets world. Coordination must emerge through rhythm and mutual awareness. Management must cultivate conditions rather than command outcomes. Identity must be enacted through practice rather than declared through statements. This is not rejection of Beer's insights but their evolution. Organisations still need operations, coordination, regulation, intelligence, and identity. But these are not separate systems to be optimised independently. They are aspects of a single living process, distinguishable in analysis but inseparable in practice. > The submarine has always been the sea, temporarily crystallized into different density but never truly separate. Organisations have always been patterns in larger fields, maintaining form through constant exchange rather than fixed boundaries. Intelligence has always been ecological, emerging from interaction rather than observation. The question is not whether to recognise this but how quickly organisations can align their practices with reality. Those that thrive in accelerating change will be those that recognise themselves as participants rather than observers, as patterns rather than entities, as becomings rather than beings. > The periscope must come down, not because observation is unnecessary, but because it perpetuates a false separation that prevents genuine adaptation. The future belongs to organisations that can feel currents through their entire hull, adjusting continuously to forces they simultaneously create and encounter. We are not discovering something new but acknowledging what has always been true. The hand has always been in the water, shaping and shaped by its currents. The market has always been performed through engagement, not discovered through research. Adaptation has always happened through participation, not observation. The organisations navigating the rapids ahead won't predict the future but participate in its emergence. They won't control their operations but cultivate conditions where excellence emerges. They won't declare their values but embody them in every tool and process. They won't stand apart to observe but dive deeper to engage. > They won't stand apart to observe but dive deeper to engage. The sea is already inside the submarine, in the moisture of breath, the salt of blood, the pressure in every pipe. We are not separate from what surrounds us but temporary crystallizations of it. The question is whether to stop pretending otherwise. > The sea is already inside the submarine, in the moisture of breath, the salt of blood, the pressure in every pipe. --- ## The Architecture of Not-Yet *Why systems fray, adapt, resist imposed clarity, and have to be understood from within.* Canonical URL: https://morenou.se/blog/architecture-of-not-yet/ Type: Essay Published: Nov 23, 2025 Subject: Systems Keywords: Systems, Complexity, Organisational Design, Emergence, Intervention Audio version: 18:54 (linked from the article page) ### 1. The Loop The world rarely breaks all at once. It frays. > The world rarely breaks all at once. It frays. Systems don't announce their transformations. They shift beneath notice, rewriting their own logic while we're still operating from yesterday's map. What looks like dysfunction is often the system's intelligence at work: adapting, responding, reacting to your reactions. A small decision cascades through invisible networks. A shift in tone becomes a new equilibrium. The system begins to vibrate at frequencies no dashboard captures, until the original shape is lost, and no one can quite remember what "normal" was supposed to look like. This is not a malfunction. This is not a disorder. This is what systems do. The more you try to freeze them in place with metrics, models, and fixed roles, the more they slip sideways, quietly rewriting themselves beneath the surface. > We are not managing machines. We are dwelling inside patterns that think back. ### 2. What We're Actually In Most of the time, when we talk about "systems," we're lying to ourselves. We say "system" and think of something tidy. Something we can map, model, control. But real systems, the ones we live and work inside, are not designed. They emerge. They evolve through habits, assumptions, and the thousand tiny decisions no one wrote down. They are not engineered. They are grown. > They are not engineered. They are grown. And that's what makes them hard to see. Because they don't announce themselves. They don't hand you a diagram and say, "Here's how I work." Instead, they surface through tensions. Through contradictions. Through that uncomfortable sense that what's supposed to work… doesn't. You try to improve a process, and people resist. Not because they don't like change, but because you didn't see what the process was holding together. You reorganize a team, and performance dips not because the structure was bad, but because it disrupted an invisible agreement, a social rhythm you never knew was there. [Maurice Merleau-Ponty](https://plato.stanford.edu/entries/merleau-ponty/) reminds us that perception is not detached observation; it's participation. You can only understand a pattern from within, through your body, your relationships, your timing. Systems are the same. They're not watched. They're inhabited. > They're not watched. They're inhabited. And when we pretend we can fix them from a distance, we flatten what's alive into something dead. ### 3. Why Most Attempts to Intervene Fail > Most interventions fail because they are not interventions. They are impositions. Someone draws a target. Someone else draws a plan. A model is selected, frameworks are invoked, and then something is rolled out as if systems are waiting for instruction. As if they are machines in need of better input. As if you can plug a new process into a human network and expect behavior to change on cue. But systems don't respond to intention. They respond to conditions. They shift only when the forces that shape them are acknowledged: forces that don't live in spreadsheets or org charts, but in rhythms, frictions, expectations, and accumulated distrust. We say "this strategy didn't land." But land where? We imagine a flat surface, but real systems are textured and layered. Tilted by history, warped by pressure, and often protected by patterns that exist for reasons no one can fully explain. Systems preserve themselves even against our best efforts to improve them. [Gregory Bateson](https://en.wikipedia.org/wiki/Gregory_Bateson) warned that information isn't data; it's "the difference that makes a difference." And most of the time, our efforts fail not because we didn't do something, but because we couldn't sense what mattered. This is not resistance. It is intelligence. It's the system's way of saying: You didn't understand what I am. > This is not resistance. It is intelligence. ### 4. The Reflex for Clarity You might be asking yourself, "Okay… but what do I actually do with this?" And that's a good question. It's a human question. It comes from the desire to move, to contribute, to act. But instead of answering it directly, we want to do something that may feel less comfortable. We want to stay in that space, just for a moment longer. Not to delay action but to understand the kind of action this system really requires. Because sometimes the urge to act comes too soon. Sometimes it's not driven by insight, but by discomfort. The discomfort of ambiguity. The anxiety of not knowing. The inherited pressure to "solve" before we've fully seen. So before we move, let's pause. Let's lean closer to that discomfort. Let's ask why so many of us have been taught that clarity must always be immediate. ### 5. On the Urge for Clarity Some will say this isn't clear. That it needs to be simpler. More actionable. That we should translate the insight into steps, next actions, bullet points. But sometimes, that urge isn't really about clarity. Sometimes, it's about relief. The relief of being told what to do. The comfort of direction. The safety of being guided out of ambiguity before it has a chance to ask something deeper of us. This is understandable. Especially in fast-paced environments. Especially when the stakes are high, and the pressure to produce is real. Thinking slowly can feel like a risk. But it's worth asking: When did thinking become a liability? When did the absence of instant answers become a threat? What if clarity isn't what's missing? What if what's missing is our willingness to sit in the discomfort of not knowing just long enough to notice what we've been skipping over? Because clarity is not always something you're given. Sometimes, it's something you earn. > Because clarity is not always something you're given. Sometimes, it's something you earn. ### 6. Seeing Like a System At some point, if you're paying attention, your vision begins to change. You stop asking, "What's broken here?" and start asking, "What is this trying to hold together?" You begin to notice relationships before roles. Movements before milestones. You stop seeing isolated events and start sensing patterns: the recurring tensions, the subtle feedback loops, the way every solution you've ever launched had a shadow it quietly cast. Systems thinking isn't just another lens; it's a new way of inhabiting the room. It's what happens when you stop standing above complexity and start feeling it as something you're inside. You become less obsessed with answers and more attuned to forces. Less focused on outcomes and more curious about dynamics. To see like a system is not to map everything. It's to realize you can't. You will never capture the whole thing. That's not a failure. That's a fact. What matters is whether you're willing to stay in a relationship with what you don't fully understand. As Merleau-Ponty would insist, perception isn't about looking from outside; it's about finding yourself already entangled. Systems are lived-through, not looked-at. > Systems are lived-through, not looked-at. This is not soft. This is not hand-wavy. This is the kind of attention that makes strategy real. ### 7. When Expertise Isn't Enough It's easy to assume that experience equals understanding. That working with top organizations, mastering business models, delivering outcomes: that these things grant us insight into systems. But systems thinking isn't a layer of polish. It's not a methodology you apply after the strategy deck is built. It's a way of perceiving that changes how you frame everything else. And sometimes, even the most confident facilitators, those who've led change across industries, fall back on tools that were never designed to hold complexity. They use structures where reflection is needed. They demand clarity where coherence hasn't yet emerged. They measure what doesn't want to be counted yet. Ivan Illich once warned that tools become dangerous the moment they begin shaping the user's thinking instead of being shaped by it. That's what happens when we bring the wrong frameworks into living systems. The tragedy isn't that they're wrong. It's that they're rushing. And when speed replaces understanding, systems don't shift. They snap. > And when speed replaces understanding, systems don't shift. They snap. ### 8. When Structure Interrupts Understanding It started with something honest. A conceptual frame grounded in reality: anchored in data, shaped with care, framed in a way that makes complexity digestible without pretending it's simple. It wasn't chaotic. It wasn't unclear. It was emergent, something that needed time to take shape through conversation and reflection. But instead of staying with it, the process was pulled downward. Forced into a predefined structure. "Split it under these headers." "Break it down into components." "Make it fit the template." And suddenly, what was once coherent became disjointed. And what was once alive became a list. > And what was once alive became a list. Then came the dissection. Leadership asked teams to quantify what had never been framed that way. Asked to reduce it so it could be "measured" not in service of learning, but to fulfill the ritual of looking strategic. And the worst part? Those who disrupted the coherence now frame the confusion as evidence of someone else's failure. ### 9. Skipping the Middle Sometimes the problem isn't that the work is too conceptual. It's that the system no longer knows how to handle concepts. Someone presents a clear, grounded insight and the first reaction is: "This feels vague." "Where's the action?" "Is this tactical enough?" Instead of working through the concept, turning it, deepening it, testing it, the conversation leaps to execution. To movement. To artifacts that look strategic. But the issue isn't the idea. It's the refusal to stay with it. To think through it, not around it. To sit inside its shape long enough for real implications to emerge. You can't demand action from what you haven't made space to understand. > You can't demand action from what you haven't made space to understand. ### 10. Acting Without Oversimplifying > The temptation to simplify is everywhere. Especially when the stakes are high. Especially when people are watching. Complexity feels like a threat: something to shrink down, to decode, to control. So we compress it. Turn it into a bullet list. Reduce it to a dashboard. Make it look neat enough to act on. But action based on simplification isn't a strategy. It's performance. It gives the illusion of momentum, while the actual system continues to behave in ways no one understands, least of all the people pushing the levers. What systems thinking teaches, and what most strategy avoids, is that clarity doesn't always come first. Sometimes, you have to move with partial understanding. We act, listening carefully, like people who acknowledge we might be wrong. This is what Donna Haraway might call staying with the trouble: not solving it from above, but entangling with it from within. Not simplifying it to escape complexity, but learning to live responsibly inside it. You stop looking for the one right move. And instead, you ask: What kind of moves does this system invite? What kind does it resist? ### 11. Designing for Emergence Most designs assume a fixed destination. A goal is defined. A plan is built. Steps are laid out in sequence. Success is measured by how closely reality follows the script. But real systems don't follow scripts. They evolve. They shift under observation. They respond to pressure in ways you can't map in advance. This is what it means to design for [emergence](https://en.wikipedia.org/wiki/Emergence). You don't control the outcome. You tune the conditions. You create boundaries, not cages. Rhythms, not rules. Incentives, not mandates. > You create boundaries, not cages. Rhythms, not rules. Incentives, not mandates. Designing this way requires a different kind of courage. The courage to be surprised. The courage to let something better than your plan emerge, and to recognize it when it does. You're not designing a solution. You're designing a situation. And in that loop, something becomes possible that no static plan ever could. > You're not designing a solution. You're designing a situation. Alfred North Whitehead called this the fallacy of misplaced concreteness: when we mistake our models for the thing itself. Systems thinking begins the moment we stop confusing our plans with the process. ### 12. On the Feedback Loop You work with tension, not against it. You build forms that you can learn. And learning doesn't happen through instruction. It happens through feedback. Not metaphorical feedback. But physical, relational, lived. A sculptor shaping clay. A coder testing output. A designer watching a user hesitate. A team sensing friction before it's spoken. Gregory Bateson called this difference that makes a difference. And that's the point: feedback doesn't just inform; it shapes. It becomes the system. You're not laying tracks. You're walking on a riverbed. The ground will change because you stepped there. > You're not laying tracks. You're walking on a riverbed. The ground will change because you stepped there. ### 13. When the Frame Is Already Broken Sometimes, confusion isn't a lack of clarity. It's the result of a frame that never held together in the first place. We ask people to analyze what was never defined. To dissect complexity using tools meant for clean separations. To quantify ambiguity, then somehow synthesize it into a story we can feel good about but not too deeply. People aren't resisting thinking. They're reacting to broken framing. Facilitation becomes performance. Discussion becomes posturing. And the work, whatever it was trying to be, is lost under the pressure to appear productive. Systems don't care about appearances. They respond to coherence. ### 14. Being Inside Doesn't Mean You're Blind It's tempting to think that insight requires distance. To really understand a system, you have to step outside of it. But sometimes, stepping outside means you lose the very context that gives things their shape. Systems aren't abstract. They're lived. They're held together by rhythms, habits, language, relationships, past decisions, unspoken rules. And these things don't announce themselves. You feel them. Over time. By being in it. Understanding doesn't come from detachment. It comes from proximity with awareness. You need context, memory, friction. You need people who know how things got this way, and why they've stayed that way. You need people who can feel what's shifting before there are metrics to prove it. Hans-Georg Gadamer called this the fusion of horizons: the moment where our own lived perspective meets the history and structure of another, and meaning emerges not by standing apart, but by standing in relation. Insight doesn't demand detachment; it demands honesty about where you stand. It requires honesty about your position: your limits, your understanding, your commitment to stay close. > Insight doesn't demand detachment; it demands honesty about where you stand. Sometimes the clearest view comes not from those outside the system, but from those who've learned to move inside it with care. ### 15. Principles, Not Answers We often end strategy with answers. Lists. Metrics. Deliverables. But real systems thinking doesn't end. It orbits. It moves through questions, returns with deeper ones, loops back again. Each insight is a node. Each node shifts the pattern. So instead of answers, we leave with principles. Not rules. Not tools. Just reminders, anchors for moving through the unknown: Listen before you move. Act like you're inside, not above. Design with feedback, not control. Name tensions, not just goals. Stay with what doesn't resolve. Whitehead might call these "lures for feeling": concepts not meant to control action, but to invite new ways of participating in it. ### 16. The Architecture of Not Yet > Systems thinking does not give you mastery. It gives you rhythm. But there's something else it teaches, something harder to accept. Sometimes, the most radical act isn't building what's possible. It's preserving space for what isn't. > Sometimes, the most radical act isn't building what's possible. It's preserving space for what isn't. We live in an era that worships the available. If the technology exists, we use it. If the framework fits, we apply it. We engineer solutions from what's at hand, then wonder why they feel like compromises. But real systems thinking, the kind that changes things, begins with refusal. The refusal to let current tools define future possibilities. The refusal to fill every void with available answers. The refusal to mistake what we can build for what needs to exist. This isn't about waiting for better technology. It's about maintaining fidelity to a structure that hasn't arrived yet. You build the scaffolding, not the building. You preserve the question, not rush to resolution. You engineer absence. A purposeful void signaling something else belongs here. > You build the scaffolding, not the building. You preserve the question, not rush to resolution. You engineer absence. [Heidegger](https://plato.stanford.edu/entries/heidegger/) called this [Gelassenheit](https://en.wikipedia.org/wiki/Gelassenheit): letting-be. Not passive waiting, but active preservation. Holding space open against the pressure to fill it with what merely works. So we return. Not because we failed. But because the system, the real one, the one that matters, hasn't fully emerged yet. And our job isn't to force it. Our job is to keep the space ready. We shouldn't build only with what exists; we must also build for what doesn't yet. Engineering should not be a mirror of available tools but a scaffolding for necessary absences. Sometimes the most important part of a system is what we leave unfulfilled, because it holds space for a future structure that hasn't yet arrived. That, too, is design. To build toward what must exist, even when it cannot yet. Especially then.