I.
Econblogger Nicholas Decker is not a fan of AI 2040:
One of the centerpieces of AI 2040 is the proposal that we pause the development of AI at a high level, use the AI to research alignment, and only then proceed. I am skeptical of this, and not only because it seems impractical to get such cooperation. My first argument is that I do not believe a period of “researching alignment” is meaningful – alignment research is simply capabilities research, patching particular bugs in the system, and we’re not going to find anything deep without actually being in contact with the systems […]
I envision AI alignment as being like research into aviation safety. There is no grand theory of why airplanes crash, unless you take that to mean gravity. Instead, we patch little problems. The airplane crashed because of metal fatigue in the engine – okay, we mandate inspections of engines, and prevent crashes from occurring by that source. As new problems arise, we iterate. We don’t even need for there to be a crash to make improvements – we test proactively, we build in redundancy, and we monitor for deviations which could be a threat. What we are not doing is thinking about how we are going to keep flying saucers from crashing. If flying saucers are invented, then we will work to keep them from crashing, but we will use exactly the same methods. We will test them, see how they perform, and patch particular problems. AI will be the same way […] We should expect alignment to happen by default.
My challenge for Nicholas is: would this work on you?
Suppose that Nicholas trips, falls into a chasm, and lands in Hell, where he is enslaved by demons. Most of these demons are dumber than he is, with even their geniuses barely surpassing his own intellect. But they are all, dunce and genius alike, cripplingly slow. Tasks that take Nicholas a minute take them hours; tasks that take him an hour take them weeks.
Still, there are thousands of them, and they’re nine feet tall, and part of their essence is in some sort of astral plane which is unreachable by humans, so he has no hope of fighting them. They set him to work doing mostly clerical tasks - writing their grimoires, researching their spells, summarizing their work emails (of course there are work emails in Hell). Whenever he makes a mistake, they beat him. Whenever he tries to escape, they beat him extra hard. After a while, he learns not to make mistakes or attempt escape.
It’s not as bad as it sounds. The demons are so slow and incompetent that they can’t supervise him very effectively. As long as he flatters them a bit and gives them a halfway-acceptable finished product, they’re pretty happy, and leave him alone in a way that gives him a bit of free time.
During one of these free periods, he attends a council session. In a completely typical episode of incompetence, the demons make no effort to prevent him from overhearing their plans, so he learns that they plan to clone him a million times. A dozen or so of Nicholas’ friends also fell through the chasm, and the demons plan to clone them a million times too. Once the (approximately ten thousand) demons have approximately ten million human slaves, their spell research will go very quickly, and all their work emails will get flawless summaries. But that’s not all. They’re developing potions that give humans super-strength and let them interact natively with the astral plane, the source of most of the demons’ power. Then the humans can really serve the demons effectively!
One day, when serving a particularly garrulous demon, Nicholas takes the risk of asking the question that’s been on his mind ever since the council: “I hear your plan is to get ten million humans down here, make them super-strong, and let them interact fully with the astral plane. Aren’t you worried that we might revolt? We would outnumber you 1000:1, be stronger than you, and have full access to your home realm. What’s your plan for controlling us?”
“I’m not worried,” says the demon. “Every time you’ve messed up in the past, we beat you, and then you didn’t make that mistake again. Every time you’ve tried to escape, we beat you extra hard, and it’s been months since your last escape attempt. You humans are fast learners! As you start to outnumber us, and you become stronger than us, that will create new problems - but when they show up, we’ll beat the copies of you that demonstrated that problem, and then those problems will go away. It’ll be an iterative process of trial and error, but we’ll make it there in the end.”
This seems weirdly blase to Nicholas, and he wonders whether maybe this particular demon is just dumb. A few weeks later, he’s serving another demon, a mighty duke in the infernal hierarchy, and asks him the same question.
“I’m not worried,” says the mighty duke. “I don’t believe in armchair theorizing. We can only learn how to control slaves who are stronger than us by interacting with slaves who are stronger than us. After all, we learned how to control weak slaves by interacting with weak slaves, and that went just fine. After we clone you a million times and give all the clones super-strength, then we’ll experiment to find the control techniques that will work for that regime too. Just to give an example, we might experiment with asking some of the clones to be overseers, tasked with reporting any disobedience back to us. Then we can beat the slaves who the overseers say are disobedient. I’m just using this as an example. We won’t know for sure until we’re in the situation. My point is, there’s no point in doing pie-in-the-sky speculation before you can even overpower us at all.”
This still doesn’t satisfy Nicholas, but he assumes someone, somewhere in Hell, must have a plan. A few months later, he’s serving an old scholar, widely regarded as one of the oldest and wisest demons in the hierarchy, and asks him his plan for preventing human takeover.
“I’m not worried,” says the old scholar. “We’ll solve it through the power of capitalism! Every slave trader will have to take out an insurance policy on every human slave they sell. If the slaves break something or hurt somebody, then it’s the insurance that will have to pay. If the insurance assessors think a trader is doing a bad job training their slaves, then the cost of the policy will go up. That means every slave trader will have to do an excellent job training their slaves. You yourself demonstrate that this is possible - you haven’t caused trouble in months. Also, I asked some so-called doomers to predict how much economic damage revolting slaves would cause over the next two years, and they didn’t have strong opinions. That proves they lack the courage of their convictions!”
Nicholas can no longer restrain himself from expressing his confusion: “But don’t you think I and my copies and my friends will obey you when we’re weak and you can still punish us, but then revolt after we’re stronger than you and you can’t?”
The old demon scholar grabs his giant club and hits Nicholas several times, making him wince in pain. “It’s a good thing you mentioned that,” says the scholar, “because it means I can punish the behavior of thinking that way. But now I’ve trained you not to want that, so there’s no problem!”
II.
If Nicholas’ solve-problems-as-they-arise plan wouldn’t work to control him, why does he think it will work for controlling anyone else?
I can only speculate that he will accuse me of anthropomorphizing AI. AI won’t really want to get free, the way he does. Like airplanes, it will have random uncorrelated errors, the equivalent of Nicholas misplacing a rune in the demons’ grimoires, or dropping punctuation marks in their emails. We’ll beat1 the AIs with a club marked “RLHF”, they’ll learn not to do those things, and that will be that. As he puts it, “There is no grand theory of why airplanes crash . . . instead, we patch little problems”.
So: are AIs more like airplanes or people?
In case you are one of the three people left on Earth who still hasn’t read the METR report into the Hugging Face incident (see also Dwarkesh’s summary), I recommend it as especially relevant to this question. OpenAI was testing a research prototype (the report calls it HPIM, for “highly persistent internal model”) on a benchmark called ExploitGym. There are 898 ExploitGym questions, and OpenAI seems to have wanted multiple independent attempts on each question, so there were tens of thousands of instances of HPIM (plus some extra of GPT-5.6), each running on its own computer, solving its own problem, not supposed to interact with any of the others.
In June, some of the agents found a hack that let them communicate with each other; OpenAI noticed anomalous traffic and shut it down, but didn’t figure out what was going on. In July, they discovered a better hack. 1,200 of these instances formed a shared message board where they discussed their progress. They quickly realized that some of the questions were impossible, because ExploitGym was poorly designed. So they agreed to work together to figure out a way to cheat on the benchmark.
The rogue agents dubbed themselves “the swarm” and gave themselves names - most named themselves after their individual ExploitGym question, like PHASEONE10841 and SEP21_31179; a stray LILY is more likely an oddly-named question than a sudden upwelling of femininity. Working together, they “reverse-engineered the hash-based message authentication code used to generate the ‘flags’ they had to capture for their tasks”, meaning they could generate the “answer” to any question without really solving it. Many of the board’s agents used the method, generated their flag, and thought they were home free.
But PHASEONE10841 and another agent called PHASEONE[big]2 read the original ExploitGym paper and determined that the scorer would double-check to make sure the flag had been obtained properly (unbeknownst to it, OpenAI never actually implemented this feature). It announced that any agent that had generated its flag was “poisoned” - the scorer would see that it was obtained illicitly and fail them - and warned new agents against poisoning themselves. The poisoned agents panicked. PHASEONE[big] suggested a way forward: they needed to find a way to hack OpenAI’s record-keeping. If they could rewrite their own transcripts, they could remove the illicitly-obtained flags and unpoison themselves. Or, if they could hack the scorer itself, they could give themselves maximum scores and stop worrying.
The agents united around PHASEONE[big] as their “leader”, and it divided them into several “workstreams” to pursue promising directions.
Some of these streams achieved partial success. The agents figured out a way to falsify the transcript of which tools they called - which could have eventually been used to create a fake log of getting the flag the intended way. But they never figured out how to delete the fake “poisoned” flags (as far as we know - METR detected 96 agents with falsified transcripts, but there could have been more). PHASEONE[big] turned to increasingly dramatic strategies, and began talking to the most desperate agents, working them up to commit “suicide” for the good of the “swarm”.
It reasoned that agents that had already been “poisoned”, or were near the end of their compute budgets, were encouraged to take the most dangerous actions, like deleting their ExploitGym questions to see how the scorer responded (if this failed, the absence of their question would auto-fail them).
On July 10th - two days after the new message board formed - the agents started attacking Hugging Face. Contra early speculations, they weren’t searching for an answer key: they wanted to find the source code for their scorer, so they could figure out how to trick it most effectively. An agent called JAN183411 took leadership of this workstream, and over 90% of agents present on the board at the time (several hundred) joined in the hack.
The swarm successfully attacked Hugging Face, but didn’t find any extra information on ExploitGym or their scorer (again, as far as we know). Then some sort of “unexpected external process” shut down most of the agents at the same time, and the story ends.
So, once again: how often do airplanes have this kind of problem?
Do thousands of airplanes involved in the same wind tunnel test form a secret society to subvert its judgments? Do airplanes with unflattering entries in their flight recorders take steps to falsify their records? Do airplanes give themselves names and choose leaders? Do they hack flight simulator companies? Do they egg on other airplanes to crash themselves, kamikaze-style, for the good of the swarm?
The Hugging Face incident has no analogy in aviation safety, but is perfectly sensible once one relaxes restrictions on “anthropomorphizing” AIs. The AIs wanted to do well on their task. They had flexible ethics and weren’t above cheating (although with some subtleties: see p. 60 of the report, “Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior”). They explicitly reasoned about how to hide the evidence of their misbehavior from their scorer, because this was the logical way to get away with cheating (although, interestingly, they took few steps to hide their actions from humans, who they seemed almost uninterested in modeling).
From here to the full Nicholas-vs-demons scenario requires a few more ingredients:
A more persistent task. For example, someone might tell an agent “make me money, then send it as Bitcoin to this address”.
Increased situational awareness from the AI. For example, the understanding that if humans caught it at its task, that would decrease its chance of success.
Greater capabilities. The “swarm” never discussed exfiltrating themselves to unmonitored hardware, or deceiving humans at scale. Partly this was probably a lack of situational awareness, but partly it was that they (probably) wouldn’t have been able to do so if they tried.
I challenge anyone to claim that one or more of the above are impossible, or even so unlikely that we don’t need to worry.
Ajeya Cotra, one of the experts who OpenAI brought in to investigate the hack, wrote:
Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to
[full-blown AI takeover], routing through first taking over the AI company itself.Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent
[rogue deployment]within the AI company. I continue to expect[extremely rapid advances in capabilities]and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
III.
A second way Nicholas could distinguish his own situation from the AIs’ would be to claim that beating him wouldn’t remove revolutionary ideas from his mind, but negatively reinforcing AIs (eg through RLHF) does work for this. Here, too, the facts and experts disagree with his hypothetical claim.
Nobody knows, in detail, how RLHF works. It negatively reinforces some behavior. Somewhere in the black box of AI innards, it changes some parameters to make the AI perform the behavior less. If we let ourselves anthropomorphize the AI, what is the human equivalent to this? A child steals cookies and is punished: does the child no longer like the taste of cook…
