*How well does this hold up after a year and a half, after the Year of Claude Code? *
Well, my warning that anthropomorphizing is dangerous has held up. My description of what these systems are has not. What changed during the Year of Claude Code is that the model now runs in a loop that checks its work against the gate of having to produce “running code”. Thus here we have a recent exchange between me and Claude Code:
Today's mistake, step by step:
13:26: you told the HermesInteractive agent: "reviewed …/2026-10-22-notes". It then read the review sheet and asked you which action items to keep ("keep 1 2 3" or "keep none").
13:37: you answered "2 and 3 are keepers", meaning action items 2 (call Chris) and 3 (ask Brian about his Trojan-Horse thing).
HermesInteractive, on the gemma4:26b-mlx model, took "2 and 3" to be numbers of next-run-pile numbers. So, instead, it ran:
"next-run-pile done 2": refused, because there is no Next item 2;
"next-run-pile done 3": succeeded, and closed next-item 3, the Katie Martin and Rob Armstrong piece, which you hadn't touched.
It then told you an invented story about why item 2 had failed. Your two action items were never kept.
I found it about 20 minutes later while checking its logs, and repaired it:
next-run-pile item 3 is back on the pile.
The transcript is marked reviewed, and your two keepers are in the tickler for Fri 30 Oct 09:00.
The HermesInteractive agent runs the gemma4:26b-mlx model, which made today's mistake, so it no longer gets the last word. A watcher script checks every reply a minute later. If the reply wasn't really carried out, the script does it and tells you "Reply watcher (a script, not the model): …". It runs in the existing once-a-minute job; nothing new to install.
*This is half a simulated neural network calculating and comparing embedding vectors and making temperature-zero model next-token predictions, and half Deep Blue-like walking down the game tree of a limited set of possible actions (chess moves or programming commands) and assigning values to each state reached, and then maximizing. *
It is very impressive: both that something as small as google/gemma4:26b-mlx can, properly harnessed, remember as much as it can of what it is supposed to do, and that anthropic/claude-opus can then catch actions at variance with MAMLM reports.
Don’t get me wrong: *The base model is still roughly a kernel smoother. *
*Pretraining is still next-token prediction over a corpus that is sparse in the space of possible prompts. *
*Shalizi’s “you can do that with just kernel smoothing!?” is still the right reaction to that part of the system. *
*And the “Machiavelli” failure still happens wherever nothing checks the output. *
*Plus, without outside feedback, models still state wrong answers in the same confident tone as right ones. *
“I want the why, not the how” is still a fair complaint.
However: “Prompt → continuation” no longer describes the product. Claude Code is that function called hundreds of times in a loop. Between calls, a compiler, a test suite, a shell, or a browser tells it whether the last step worked. A kernel smoother plus a verifier plus iteration is a different kind of object from a kernel smoother alone. And that means that my “not for truth-seeking” is now much too strong. Whenever the answer can be checked (code runs, a URL loads, a proof verifies), these systems now seek truth reliably.
So:
*I need to keep “not brains” and drop “not reasoning”**These things have an ability to reason built from interpolation plus outside checking, which is unlike a human mind and yet still powerful.**My parenthetical “isn’t my brain also a neural-network flexible function… trained on a data corpus that is sparse?” aside looks, eighteen months on, like it is doing as much work as my main argument.**Real disagreement is over how complex the function is and what feedback trains it, not over whether it is a function.**Yet the capability is uneven, not general: agents can refactor a large codebase and still botch a citation: they are strongest where training data and checkable rewards are dense, weakest in the long tail.*The 2026 version of “LLMs need curation” is that agents need tests, sandboxes, and review of their changes: the human checks the work, not every keystroke, and hat is a real change in the division of labor.
This is not an annoyance, but rather a cavil—for I think that the very sharp Scott Cunningham has gone down the wrong track here:
Scott Cunningham: Inside the "Brain" of Claude <[https://causalinf.substack.com/p/inside-the-brain-of-claude]>": ‘Modern AI models like Claude…. When asked to write a rhyming poem,.. it activates features representing potential rhyming wordsbeforebeginning the second line… "grab it"… "rabbit" and "habit". It then constructs the second line to lead naturally to the planned rhyme…. Claude doesn't generate text “one token at a time”—it's activelyplanningfuture content and working backward to create coherent structures…. LLMs are performing… abstract reasoning, planning, and metacognition…. Two years ago, I would’ve thought that was impossible…. Large language models are complex systems with emergent properties—much like biological organisms…
I think Scott goes awry in word 3 of his title “Brains”. In my view, LLMs are still much too simple for words like “reasoning”, “planning”, and “metacognition” to be useful terms to apply—even purely metaphorically—to their behavior. It leads to much more insight, I think, to start from recognizing that trained neural networks are extremely flexible interpolation functions from a domain (in this case, a string of words) to a range (in this case, the continuation words). They have a training data set that is sparse in the domain—unless a string of words is a quote, a cliché, or boilerplate, by the time the string reaches twenty words long there is less than an 0.1% chance that it has ever been written down before. For prompts in the training dataset your flexible function is obvious: you simply return the continuation.
But for everything else you have to interpolate, somehow. The Deep Magic of the LLMs is in that interpolation process and in the shape of the training data. And thinking “BRAINS!!!” actually, I think, makes it harder to gain insight into why they behave the way they do. All that Scott has to say in his exposition of Lindsey et al. does say interesting things about the “how”. But I want to know the “why”. And even the statements and findings about the “how” are, I think, corrupted into near-uselessness because of the “BRAINS!!!” frame.
Why do I think this?
Well, yesterday I had a University of Chicago citation:
Machiavelli, Niccolò. 1513 [2008]. “Letter to Francesco Vettori,December 10, 1513”. InThe Prince. Trans. & ed. Peter Bondanella. Pp. 109–113. Oxford: Oxford University Press.
And so I gave ChatGPT a task:
Q: Please get me the book from archive.org, and spell out the URL please.
My question asks ChatGPT to find a string of symbols, beginning with “https://archive.org/”, such that when these symbols are entered into the address bar of web browser and the “return” key is pressed, the web browser loads a file that is a scanned-and-OCRed digital version of the print book published in 2008 by Oxford University Press that is the version of Machiavelli’s Il Principe translated and edited by Peter Bondanella.
The correct string of symbols to return to accomplish this task is “https://archive.org/details/niccolomachiavel00nicc”.
But this is what ChatGPT gave me when I assigned it the task:
A:Certainly! TheLetter to Francesco Vettori, dated December 10, 1513, is included in several public-domain editions of Machiavelli's works, many of which are available at.[Archive.org]One widely available English translation that includes the letter is:
"The Historical, Political, and Diplomatic Writings of Niccolò Machiavelli"
Translated by Christian E. Detmold, published in 1882.
Volume 2 includes Machiavelli's letters.Here is the direct Archive.org link to
Volume 2, which includes the Vettori letter:URL (spelled out):https://archive.org/details/historicalpoliti02machuoftYou can find the letter to Vettori in this volume, usually around the first few pages—it’s often included in the section that discusses
The Prince, since it references Machiavelli's work on that text.Let me know if you’d like a specific page number or a passage quoted!
Yes. It did return a string of symbols beginning with “https://archive.org/”. But “historicalpoliti02machuoft” is not “niccolomachiavel00nicc”.
When I gave ChatGPT this task, I was using it as a natural-language interface to the largely unstructured data store that is the Internet. This natural language interface is extraordinarily metaphorical. Consider the little words "get… book… from…" in my prompt. I am analogizing websites—the “https://archive.org/” part of the symbol string that ends .org/—to houses with addresses. I am analogizing the rest of the symbol string that I am asking for—the “niccolomachiavel00nicc” part that follows .org/—to an object that is in the house. And I am analogizing ChatGPT’s returning the symbol string to its (a) finding the house, (b) going into the house and getting the book, and then (c) bringing the book out of the house and delivering it to me. That ChatGPT is capable of taking the little words “get… book… from…” and acting so as to construct a symbol string that is also really and truly, in a sense, getting the book from where it was—that is truly amazing. As Cosma Shalizi says:
Cosma Shalizi: "Attention", "Transformers", in Neural Network "Large Language Models" <[http://bactra.org/notebooks/nn-attention-and-transformers.html#just-kernels]>: ‘“You Can DoThatwith Just Kernel Smoothing!?!… The incredibly impressive engineering accomplishment ofmaking the blessed thing work. A[large, able and confident group of people pushed kernel-based methods for years in machine learning], and nobody achieved anything like the feats which modern large language models have demonstrated. The reason I put effort into understanding these machines and papers is precisely because the results are impressive!…
It is doing something, and something extremely impressive, but that something is not thinking, at least not as we know it. (But then, again, do we really think as we know it? We think we do. But that is all that we can really say, isn't it?)
We know that that something is not thinking, that it is not understanding and then acting but rather doing something else, because it brings me the wrong book. Or, rather, it would bring me the wrong book if it brought me a book at all. For it in fact does not bring me a book. Rather than what I wanted—the left—the web browser served up the right:
That is a failure: it has no idea that it has not accomplished its task at all. It is interesting that it (a) tried to bring my the wrong book, (b) failed, and (c ) did not know that it failed. And I am thinking about why it did what it did. But after being primed by that failure experience, I was ready to cavil at Scott Cunningham’s blog post, because its “BRAINS!!!” frame provides no way to even think about how such a set of errors might be a possible thing an LLM might do.
And that is why I think the extremely sharp Scott Cunningham is going down a currently unprofitable rabbit hole here:
Scott Cunningham: Inside the "Brain" of Claude <[https://causalinf.substack.com/p/inside-the-brain-of-claude]>: ‘How researchers are dissecting large language models to understand what makes them tick: This substack is a summary of "[On the Biology of a Large Language Model" by Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, et al. (2025)]which I am teaching today and I wrote up the lecture notes….Modern AI models like Claude are massive and messy inside—individual neurons often serve multiple unrelated purposes. To overcome this, the researchers: 1. Built a more interpretable "replacement model" that approximates Claude's thinking. 2. Traced how information flows through this model. 3. Conducted "intervention experiments" by turning specific features on/off to confirm their role…. When asked to write a rhyming poem,.. it activates features representing potential rhyming words
beforebeginning the second line… "grab it"… "rabbit" and "habit". It then constructs the second line to lead naturally to the planned rhyme…. Claude doesn't generate text “one token at a time”—it's activelyplanningfuture content and working backward to create coherent structures….LLMs are performing surprisingly sophisticated computations internally, including abstract reasoning, planning, and metacognition (thinking about their own knowledge). Two years ago, I would’ve thought that was impossible, even at whatever level this software operates at, so this alone is a learning curve idea for me…. Large language models are complex systems with emergent properties—much like biological organisms, hence the title of the paper….
This paper and its findings are being applauded because it represents a significant step forward in AI interpretability, but it's only the beginning. The authors acknowledge… limitations… including: difficulty tracking “
[attention]” mechanisms; challenges in understanding global patterns across different inputs; the complexity of the graphs, which often required significant simplification and manual interpretationFuture work will likely build on these methods to create even more comprehensive views of AI cognition, potentially leading to safer, more reliable, and more truthful AI systems. But it’s interesting how primitive we are now even with what would seem like sophisticated NLP and machine learning tools in our toolbox…
Scott is, I think, making a very normal error here. Since the rise of large language models (LLMs) like GPT-4, popular discourse has often been captivated by the notion that these systems are “BRAINS!!!”—that they think, reason, or understand like humans do, or at least that human-thought metaphors are useful ways of gaining insight into them. I, by contrast, have become convinced, because of things like the ChatGPT error above, that by far a more accurate and useful framework is to understand LLMs as flexible interpolative functions from prompts to continuations. That better captures both how these systems work and what they can (and can’t) do. Plus it has,as Henry Kissinger said, the additional advantage of being true. And so it grounds our understanding in the mathematics and engineering of machine learning, rather than in anthropomorphic metaphor.
At the core, LLMs are statistical pattern recognition engines trained on massive amounts of text data. During training, the model is given sequences of text and learns to predict the next word (or token) in the sequence. But it does not “understand” the text in any semantic sense. Instead, it adjusts billions of parameters to minimize prediction error across vast data corpora, learning the statistical regularities in the training data.
The result is a function: given an input (a prompt), it generates a plausible continuation based on patterns it has seen. This is interpolation—filling in the blanks based on context. And as these models have become larger and been trained on more data, the sophistication of this interpolation has improved dramatically. But fundamentally, this is still what the model is doing: mapping prompts to continuations.
And the power of LLMs is through their ability to do a very sophisticated kind of interpolation that we do not really understand well at all. They have a training data set that is sparse in the domain—unless a string of words is a quote, a cliché, or boilerplate, by the time the string reaches twenty words long there is less than an 0.1% chance that it has ever been written down before.
For prompts in the training dataset your flexible function is ob…
