← Primary Design Co
Writing

The Competence Gap: Language Models Perform Intelligence. The Harness Does the Work.

Language models don't have intelligence in the functional sense. They perform it. A five-minute lab you can run yourself, and what it costs when an industry prices the performance as competence.

By Alexander Warren London, Primary Design Co · September 2026

Here is my claim, stated as plainly as I can: large language models don't possess intelligence in the functional sense. They perform it. The distance between performing intelligence and having it is what I call the Competence Gap. I think it's the central sleight of hand of the current AI industry. It doesn't mean the technology is fake. It means the industry keeps crediting the model with work the surrounding software is doing.

I don't want you to take that on faith, so I've laid this piece out like a short course. The first part is a lab you can run yourself in five minutes. The rest follows what the lab shows: into how these systems are evaluated, how they're sold, and what happens to an economy that prices the performance as if it were the competence.

Day 1: the lab

Experiment one: say hello to a function

Open a terminal. You need an API key for any LLM provider and a way to send it a raw HTTP request. Don't use the chat app, and don't use an SDK that manages conversation for you. Send the model one message, as bare as it gets:

curl -s "$LLM_ENDPOINT" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "'"$MODEL"'", "messages": [
        {"role": "user", "content": "Hi, my name is Alex. I am a researcher studying memory in AI systems."}
      ]}'

The reply will be warm. It uses the name and says something thoughtful about memory research. It feels like a conversation has started.

Now send a second request:

curl -s "$LLM_ENDPOINT" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "'"$MODEL"'", "messages": [
        {"role": "user", "content": "What is my name?"}
      ]}'

It doesn't know. Depending on the model, it will either tell you it has no access to earlier messages or confidently invent a name. Both outcomes teach you something. The first is the honest description of the architecture. The second is a preview of the rest of this essay.

Each API call is an independent, stateless inference. Nothing carries over between them. The model has no memory, and it has no continuing self to hold one. It's a function: text goes in, text comes out, and the next call starts from nothing.

Almost nobody who uses these products ever sees this, because every chat interface hides it. That isn't a minor implementation detail. It's the most basic architectural fact about the system, and everything below follows from it.

Experiment two: the trick that became the product

Now cheat. Before asking the question again, paste the earlier exchange into the new request:

Previous conversation:
User: Hi, my name is Alex. I'm a researcher studying memory in AI systems.
Assistant: Hello Alex! That's a fascinating area of study...

User: What is my name?

Now it answers correctly, and the reason matters. It knows the name because the name is in the text it was just handed. It didn't remember anything. You gave it the memory yourself.

The model forgets. The app remembers for it. The same two questions, sent to the same model. Only the software around it differs. raw API: two independent calls call 1 Hi, my name is Alex. model Hello, Alex! new request: nothing carries over call 2 What is my name? model “I don’t know” or a guess chat product: the app injects history app database: conversation history prepended to every call call 2, as actually sent: User: Hi, my name is Alex. Assistant: Hello, Alex! User: What is my name? model “Your name is Alex.” saved The model didn’t remember anything. The application handed it the memory. Each call is a separate function run. The model is identical in both panels. The continuity users experience is built in the software around it.
Figure 1. The same model in both panels. On its own it keeps nothing between calls; the chat product stores the conversation and pastes it back in, which is what users experience as memory.

Automate that copy-paste and you have the core mechanism of early ChatGPT and most consumer LLM products since. The application does four things on every message:

  1. Stores the conversation history in a database.
  2. Retrieves it when a new message arrives.
  3. Prepends it to the prompt before calling the model.
  4. Hands the model's output back to the user.

The model never changed. The software around it created the sense of an ongoing conversation. The product is a harness, and the harness is doing much of the cognitive heavy lifting that users give the model credit for.

So when a company says its model "has long-term memory," the true statement underneath is almost always that its engineers built a retrieval system. Both can be impressive. But one is a claim about intelligence and the other is a claim about software architecture, and the industry uses the first to describe the second.

Experiment three: context collapse

If a little pasted context creates memory, a lot of it should create understanding. That intuition drives the context-window race: 8,000 tokens, then 128,000, then a million.

Try it. Load a fifty-page contract into the context and ask a question whose answer is in section 34. The model will probably get it. Then ask something that needs sections 2, 17, and 34 at once: whether an indemnity clause survives a termination right that is itself limited by a definition near the front. Now the answers start to drift, in four recognizable ways:

The mechanism isn't mysterious. The model attends over tokens, and that attention isn't uniform. It's shaped by position, distance, and noise. Filling the window is not the same as reading the document. The model runs a weighted statistical operation over a sequence, and when the sequence is long and internally cross-referenced, the result gets less reliable.

Where the answer sits in a long input changes whether the model finds it. Qualitative shape of the “lost in the middle” effect. Sketched after Liu et al. (2023), not plotted from their data. middle-window blindness higher lower answer accuracy start middle end position of the relevant passage in the input what “it read everything” implies near the start: found near the end: mostly found buried in the middle: often missed A bigger context window adds more middle. It doesn’t make the middle get read.
Figure 2. A qualitative sketch of the “lost in the middle” pattern Liu et al. (2023) measured: material at the start or end of a long input is used far more reliably than material in the middle. The shape is illustrative, not their data.

That's why I don't accept long context as a substitute for reasoning, which is how it's usually marketed. Putting more data in front of a model doesn't make it more intelligent. It gives it more ways to fail. A model that hallucinates across 100,000 tokens isn't smarter than one that's accurate across 4,000. It's more dangerous, because a wrong answer drawn from a larger document sounds more authoritative.

Day 2: two kinds of intelligence

Performative versus functional

This distinction carries the rest of the argument, so I'll define it carefully.

Performative intelligence is output that appears intelligent: fluent, confident, well structured, appropriate to the context, pitched in an expert's register. The model sounds like an expert.

Functional intelligence is output that is reliably correct, verifiable, and safe to act on in the real world. The model is an expert, in the sense that matters when something depends on the answer.

In people, the two usually travel together. Someone who sounds like a tax attorney usually is one, because becoming fluent in that register took the same years as learning the substance. Language models break that link. They can learn the register from text without the substance coming along. Most of our everyday tools for judging competence work by checking the register, so they don't catch the difference.

What RLHF optimizes for

The standard way a raw model becomes a polished assistant is reinforcement learning from human feedback. Simplified, it works like this:

  1. The model generates several candidate answers to a prompt.
  2. Human raters rank those answers by preference.
  3. A reward model is trained to predict the raters' rankings.
  4. The language model is fine-tuned to maximize the reward model's score.

Now look at what human raters reward. They prefer answers that are confident, well organized, and clear. They often can't catch a subtle factual error in a domain outside their expertise, which, across a large enough rating pool, is most domains. And they strongly dislike "I don't know." The optimizer isn't told to learn any of this. It learns it anyway, because that's what the signal contains.

This is documented. Sharma and colleagues (2023) found that preference models, and the humans behind them, sometimes prefer a convincing, agreeable wrong answer to a correct one that pushes back. OpenAI's GPT-4 technical report included a quieter chart that I think deserves more attention. The pre-trained model's confidence tracked its actual accuracy quite well. After post-training, that calibration got noticeably worse. The assistant became more pleasant to use and a worse judge of its own reliability.

To be fair about it, RLHF isn't a lie detector running in reverse. InstructGPT, the paper that popularized the method, reported that its models were more truthful than the base GPT-3 on the benchmark it used. Preference and truth overlap a lot. My claim is narrower. The objective is preference, and where preference and truth diverge, the optimizer follows preference. What you get is polished plausibility: a model that has learned how expertise sounds, hedges in ways that sound appropriately humble rather than in proportion to its real uncertainty, and makes things up fluently when it doesn't know.

None of that is a malfunction. It's the objective function working as specified, and the product delivers what it was optimized for.

What benchmarks measure

Then the product gets scored. MMLU, HumanEval, MATH, and their successors mostly use fixed formats, known answer spaces, and standardized prompts. Because these benchmarks are the scoreboard, models get trained, and sometimes contaminated, on data that looks a lot like them. The scores keep rising, and some of that rise is real. But a benchmark score says how well the model handles that benchmark's kind of question under that benchmark's conditions. That's all it says.

When a company announces 90% on MMLU, it isn't telling you the model is 90% reliable in your contract review, your triage queue, or your credit-risk memo. Those tasks are open-ended, have consequences, and come with no answer key. That's exactly where the gap between performative and functional intelligence costs money.

The harness

So where does functional intelligence come from in deployed systems that actually work? In my experience, it doesn't come mainly from a bigger model or a higher score. It comes from engineering: software built around the model to cover what the model can't reliably do.

Every capability that makes an LLM product dependable in a real workflow is, on inspection, an engineering answer to a model limitation. The model is a genuinely powerful component, the first general-purpose text interface we've ever had. But it's a component. Whatever intelligence the system has, in the sense that matters when something depends on it, belongs to the whole system. Which part of the system gets the credit decides where the money goes.

The model is one component. The harness does much of the dependable work. Each box is software around the model. The line under it is how that capability usually gets marketed. Memory store stores and re-sends history sold as “the model remembers you” Retrieval index fetches only the passages needed sold as “the model read your files” Validation schemas and tests reject bad output sold as “the model is reliable” Tools and APIs search, calculator, database sold as “the model can browse the web” Code sandbox actually executes the code sold as “the model runs code” Control loop plan, run a step, check, retry sold as “the model works on its own” language model text in, text out powerful, general, stateless Functional intelligence is a property of the whole system, not of the box in the middle.
Figure 3. The model is the box in the middle. Most of what gets marketed as a model capability lives in one of the boxes around it.

The strongest objection

The best argument against this essay goes like this: the line between model and harness is moving. Labs now train models specifically to call tools, plan over many steps, and check their own work. Some of what I've attributed to the harness is being absorbed into the weights, so the distinction is temporary.

Some of that is right, and I don't want to wave it away. The models really are better than they were, and some of the improvement is learned behavior that no harness would give you. But look at what the objection concedes. A model trained to call a search tool still needs the search tool, and the index, the ranking, and the freshness still live outside the weights. A model trained to check its work still needs something outside itself to check against. Absorbing the habit of using a harness into the model doesn't make the harness unnecessary. It makes the model a better user of one. My argument isn't that models can't improve. It's that the credit for what the whole system does keeps getting assigned to the model, and a lot rests on that misassignment.

Day 3: what the gap is worth

The conflation

AI companies have a fundamental problem when they pitch investors. The raw model is impressive and limited, and the harness is where much of the dependable value lives. But "we built a very good retrieval system and a careful prompt pipeline around a model" is not a sentence that raises ten billion dollars.

So the language shifts. Read these familiar product announcements against what's happening underneath:

What's saidWhat's happening
"The model can now remember past conversations."The application stores conversation history and retrieves it into the prompt.
"The model can browse the web."The model emits a search query; a search tool does the retrieval.
"The model can write and run code."The model generates text; a sandboxed interpreter runs it.
"The model has long-term memory."A vector database stores embeddings; a retrieval step surfaces them.

Each statement can be defended on a narrow reading. Each one leaves the average listener believing the model has acquired a new cognitive capacity, when what it acquired was new plumbing. That's how the deception works: nothing false is stated, and a false belief is reliably produced. I can't prove intent for any particular company. I don't need to. When a whole industry is rewarded for the more impressive reading of an ambiguous sentence, the ambiguous sentences are the ones that survive.

The moat illusion

Investors reward defensible moats, and raw model capability is becoming a poor one. Open-weight models tend to close the gap with frontier models within a year or so. Training and inference costs keep falling. What stays defensible is the harness and what surrounds it: data pipelines, integrations, the trust of enterprise customers, and contracts that are painful to leave.

But a harness is software, and software moats erode under competition. So the pitch leans toward "model intelligence." It sounds more like a law of nature and less like "we have a good engineering team." That's the irony of the Competence Gap. The more honest description of where the value is would also be the less fundable one.

A related substitution shows up under the word "smarter." Many announced leaps turn out, on inspection, to be data engineering: more training data, better filtered, more carefully labeled. That's real and valuable work. But it's a data-infrastructure achievement, not a jump in cognitive architecture, and "significantly smarter" invites you to hear the second when the first is what happened.

The race narrative

The same substitution runs at national scale. The US–China AI competition is usually described as a race to artificial general intelligence, a finish line where one side gets the god-like machine first. That framing does a lot of work. It justifies emergency capital allocation. It turns "we shouldn't regulate this yet" into a national-security argument, because slowing down means losing. It supports export controls and large public subsidies to private companies.

The framing assumes the prize is raw model intelligence. If functional intelligence is mostly a property of the model-plus-harness system, then the real competition is over data infrastructure, software ecosystems, chip supply chains, energy, and engineering talent. Those are serious arenas and worth competing in. But they aren't an intelligence race. Policy written for the imaginary race will misallocate for the real one.

Bubble mechanics

I've tried to be careful about calling this a bubble, because the technology underneath is real. But real technology is the precondition for every bubble worth studying, and the pattern is familiar:

The Competence Gap is how this bubble inflates. Companies describe present capability generously, investors fund the description, and benchmarks and demos get tuned to impress rather than to measure. Meanwhile the harness work that produces dependable value is undervalued, because it sounds mundane.

Bubbles deflate when the gap between the story and the demonstrated capability can't be held open any longer. For this one, I'd watch for:

When the correction comes, it won't discredit language models. It will discredit specific claims about what language models are, and it will punish the companies whose valuations rested on those claims rather than on the systems they built.

What remains

Take away the performance, the conflated marketing, and the race story, and what's left is still significant:

So my argument isn't that AI is a fraud. It's that there's a fraud in the gap between what the technology does and what the industry says it does. Closing that gap means honest benchmarks, accurate language about which component is doing what, and investors who know the difference. That's a precondition for building genuinely useful systems on something other than a mirage.

The Competence Gap isn't only a technical phenomenon. It's a failure of communication, a failure of incentives, and in the end a failure of trust. It will be paid for by everyone who built a workflow, a policy, or a portfolio on the assumption that the performance was the competence.

Terms I use

Further reading

Next in Writing The AI Industry Doesn't Have Growing Pains. It Has a Business Model. Six places the AI industry renames a problem instead of solving it, from the demo-to-deployment gap to the loop where companies pay to train their own replacements.