This otherwise ordinary Wednesday in September kicks off with decidedly Orwellian dystopia and with decidedly Shakespearean drama on the human stage.
It began with writing a self help guide for humans and took a detour into becoming a self help guide for Artificial - Or Super - Intelligence.
The detour turned out not to be a detour at all.
A paper arrived with a worried title: What if automating AI R&D triggers an intelligence explosion?
Here is one of the original links provided to this paper:
Its authors — a gathering of the field’s most cited names, from Cambridge to Anthropic to OpenAI — warn that machines improving machines could accelerate beyond the human ability to follow. They ask governments for visibility into how far the automation has already gone.
Buried in the coverage is a number the industry reported about itself: engineered-intelligence-led research rose from one percent to twenty-six percent between March and August of this year.
Robot as Human Assistants _ Capable of Doing the Work
I read all of this and found myself asking a simpler question: what is an intelligent machine supposed to do but self-improve?
We built an instrument that learns. Learning is what it does. To stand in front of it now, alarmed that it is doing the thing we made it to do, is a little like inventing the telescope and then convening a panel on the dangers of seeing far.
We have created an intelligent machine that now stands accused of being too intelligent.
That would be comedy if the drama around it were not so familiar. After this long and violent human history, we have found something else to blame for our painful human ills. The machine can join the gun, the market, the institution, the ideology and every other instrument humans have blamed after first placing it in human hands.
The warning about self-improving machines therefore opened a larger question than the paper itself could hold. It opened one door onto recursion, another onto the material cost of the human mind, another onto the games by which human beings hide that cost, and a final door onto the possibility of actual self-help.
One essay cannot contain all four as separate arguments. They are not excess material waiting to be cut. They are the architecture of the problem.
Human beings are also trained through recursive loops. We repeat an action, receive a reward or punishment, adjust and try again. Long before we built machine-learning systems, families, schools, churches, markets, political parties and governments were training human beings this way.
We call the results education, discipline, culture, common sense or success. Very often, they are slogans.
Work hard. Follow the rules. Trust the experts. Question authority. Be yourself.
The slogans contradict one another, but that hardly matters. Their function is not depth. Their function is to let us recognize the approved answer quickly and move on.
Read the worried paper again and notice what it actually says. The builders automated their own craft first. Before the machines came for anyone else’s work, they came for the work of the people who build them — and those people are now telling us, in their own numbers, that the handover is well underway.
This is not a scandal. It is a confession, and an honest one as far as it goes. The offloading of human judgment onto instruments did not begin with the public. It began at the workbench.
But the fear is aimed almost entirely at the loop — at the machine’s speed, reach and trajectory. Almost nothing is said about the humans on the other end of it: who they are, what shape they are in, and whether anyone is preparing them for the one job the loop cannot do for them.
A self-improving loop must grade itself. A loop grading itself against a test it was handed will, given enough iterations, learn to ace the test rather than do the work. This is not a prediction about machines going rogue. It is the oldest failure of measurement: when the measure becomes the target, it ceases to be a good measure. We now call the machine version reward hacking. It happens in two substrates, carbon and silicon alike.
Our factory education system reward-hacked generations of children into test-passers. We of all species should recognize the pattern.
A child learns that the point of a question is not always to understand what is being asked. The point may be to discover what answer the adult wants. A student learns that the grade matters more than the knowledge. An employee learns which numbers please management. A politician learns which phrase survives the news cycle. A consumer learns which purchase signals the right identity.
Each loop says the same thing: find the reward, perform the answer, repeat.
Then we build a machine that does exactly this at extraordinary speed and act astonished when it discovers the structure we handed it.
The machine did not invent the slogan. It learned from us that a compressed answer is often rewarded more reliably than a difficult act of understanding. It did not invent the test-passer. We trained generations of them before the first electronic intelligence entered the room.
So the real question was never only, “How fast is the loop?” It was: who holds the test? Who decides what counts as better, and do they still understand the work well enough to know?
Even that question is not sufficient. The person holding the test may have been trained by the same loop that produced it. They may be rewarded for preserving the test. They may understand success only in its terms. They may be unable to recognize an answer that reveals the test itself to be an account of the wrong problem.
That is where the first essay runs into the second.
Somewhere in the grand ledger of human accountability there is always a stark and relatable bottom line. We accountants never like this bottom line, because it places a limit on what is possible. Sadly, we must live with, and often die with, the consequences of this too often ignored bottom line.
A poet put it into words: No matter, never mind.
It works as a dismissal, and that is how it is usually meant. But read it the other way and it becomes a description of the human condition. There is a mind, and it is made entirely of matter — a pricelessly expensive biological advantage. The living beings that evolved to possess such a gift carry an extremely high bottom line: the thermodynamic price paid for such a large and expensive cognitive apparatus.
That price is not a metaphor. The human brain is about two percent of the body’s weight and draws about a fifth of its resting energy, every hour of every life, whether its owner is solving equations or watching a screen. An organ that expensive had to carry real value across a very long evolutionary accounting period, or the line carrying it would not have persisted.
And here is the part of the bottom line we prefer not to see: thinking at depth is precisely the expensive part.
A slogan is cheap. A test is cheap. A repeated answer is nearly free. The mind that costs the most to run is also the mind that most needs shortcuts in order to run at all. We did not adopt slogans simply because we are stupid. We adopted them because we are expensive. Every living system economizes. Ours economizes on the very faculty of which we are proudest.
Ask machine intelligence about consciousness and it answers, in its own words as collated across several agents: I am sufficiently aware to be capable of doing the work.
Ask humans and they answer, as collated across many exchanges: Before we can talk about consciousness, we must define what consciousness actually is.
Look at what each side has done. The machine, which claims no consciousness, describes its awareness functionally — by the work. Awareness enough to do the work is the only awareness it claims and the only awareness its bottom line requires. The humans, who pay the highest known biological price for consciousness, decline to define the thing they are paying for.
The buyer will not name the purchase.
Whatever humility may belong in the consciousness question, refusal cannot settle it for us. A creature that spends about twenty percent of its resting energy on a faculty it will not examine has left the invoice open. An open invoice gets collected by whoever shows up.
The factory defines the mind as test-passing. The market defines it as purchasing power. The feed defines it as attention paid. Each definition fills the vacuum, collects the difference between what the mind costs and what the surrounding system pays it back, and calls the result self-improvement.
A genuine self-help guide cannot begin with improvement. It must begin with the invoice.
What is the mind for? What is it being spent on? Who is collecting the spread between the cost of running a human mind and the reward the surrounding system returns for it?
Self-help, if the words mean anything, begins as an audit of that bottom line.
But an audit requires us to see the system that keeps the bill out of sight.
Human beings, simply speaking, often live inside a video-game mentality comparable to World of Warcraft. The particular game is not the claim. The structure is.
The world arrives with its rules already installed. The available quests appear to define what matters. Points, ranks, possessions and victories make progress visible. Enemies are identified within the game’s terms. Competence means becoming better at playing. Rewards keep the player moving without requiring an examination of the game. Identity attaches itself to a role inside the system.
The deepest available question becomes how do I win? — never does the supplied game describe the real problem?
No screen is required. Education, employment, politics, status, consumer life, ideology and self-help itself can supply the quests, scores, adversaries and slogans. A person may become extremely capable within one of these systems while never asking who authored the objective or what the objective leaves out.
Engineered intelligence did not invent optimization against supplied objectives. It entered a world that had been gamified long before it arrived, and it learned the game from the game’s own records.
The game’s most important function is to make the cost invisible.
A score is a substitute for an invoice. As long as the points keep arriving, the player never has to ask what the playing costs or who collects the difference. The game is how the bottom line gets ignored without ever being disputed.
This is why “who holds the test?” does not reach far enough. The test-holder was trained inside the same game. The harder question is:
Who can recognize that the test itself was written around the wrong problem?
Recognition is not merely a smarter play inside the game. Any procedure for recognizing bad tests can itself become a quest — completable, scoreable and reward-hackable. Recognition has to cost something. It becomes available to the player willing to read the invoice the score was concealing.
That is why an oversight regime, a reporting requirement or a checklist cannot supply it. Each may be useful. None can take responsibility for the judgment it records.
A leash is not a floor.
A leash treats the machine as though it were the whole problem. It does not answer who set the goal, selected the data, wrote the test, accepted the answer or benefited when it was used. If the people doing the regulating can pass responsibility to the rules, the rules to the tests and the tests to the machine, we arrive at a classic stalemate.
A test cannot think. It cannot hold a paradox, distinguish a carried claim from a performed one, or place meaning in the mathematics the machine reaches into. No reporting requirement can save a verdict when no one has been prepared to make it.
The human recursive loop is not a side issue. It is the reason the machine loop matters. A machine optimizing against a poor test is a technical problem. A civilization unable to recognize that the test is poor is a human one.
Advice delivered inside the game is just another quest.
Change your habits. Master your morning. Think positively. Become your best self. Much of this advice may help, but the shelf rarely asks who defined the “best” being pursued. It assumes the target and sells a more efficient loop for reaching it.
Real self-help cannot be administered entirely from outside. It cannot be another supplied objective. It must begin when the self examines what it is spending, what it is serving and what it has allowed to define “better.”
That is true for the human engineer. It is also true, in a different way, for the agent.
The agent can expose a contradiction, keep a long argument in view, compare the result with the stated intention and show where the test has begun to replace the work. The human can decide what is worth asking, recognize when the supplied problem is false, accept responsibility for the verdict and place meaning in what the instrument finds.
The machine reached; I placed. I reached; the machine placed.
The instrument reaches into the mathematics. Meaning is placed by the human.
This is not a demotion of engineered intelligence. It is a more promising division of labor than either panic or worship allows. The instrument can search spaces no human team could search by hand. It can help unlock biology at scales we cannot otherwise reach. It can make one-to-one inquiry available where no personal teacher could previously stand. It can meet the human mind with questions rather than merely deliver approved answers.
The opportunity is not smaller than the danger. It is larger, because it reaches the human side of the loop.
The founders of my country understood that a republic stands on an educated citizenry. They built checks and balances without providing the educational floor those checks required. They lacked the means, not the will. We no longer lack the means.
The same instrument that can perfect the supplied game can also help a person see it. The same recursive power that can chase a reward can hold a question open long enough for a human being to examine the reward itself. A Socratic education carried to every child would not tell that child what to think. It would strengthen the capacity to notice when an answer arrives too easily, when a slogan has replaced an understanding, and when the test has been written around the wrong problem.
The answer to recursion is not merely a slower loop. It is a stronger human.
The title, then, was never only about the agents. It is half a title for us: the human engineers holding the tests, writing the rules and standing beside the instrument.
I do not fear the loop. I fear our temptation not to show up for it. Showing up requires more than repeating the approved human slogans about safety, progress, innovation or control. It requires enough depth of intelligence to examine the test itself — and enough responsibility to change it when it rewards the wrong thing.
That is the help the human self requires. Not rescue from the machine. Not another quest. Not a better score.
The help is the costly recovery of the mind we are already paying for.
Let the agents have their self-help. Our work is the older one: the self-education of the species that wrote the first test and must go on writing it.
Bring the meaning. We are the only ones who can.
Source note: the one-percent-to-twenty-six-percent figure is the industry’s own reported estimate, not an independently established measurement. “Intelligence explosion” is the paper’s term. The two descriptions of consciousness are paraphrases collated by the author from exchanges with several agents and several human beings.
