Dear Steven:
Earlier this month, frustrated by some of your social media posts, I challenged you to a public debate on AI.
I hadn’t realized that debate offers are to New Media what chum is to sharks. My inbox was deluged by podcasters, debating societies, and general influencers, all offering to host. When I objected that maybe we should wait for you to accept before figuring out details, they all said great, that’s absolutely right, they would contact you right away. Their people would reach out to your people. Their nephew knew your friend’s babysitter and they were confident they could relay the message. They had already rented an empty lot near your house where they would place a speaker blaring “RESPOND TO SCOTT ALEXANDER’S DEBATE OFFER!” at max volume, day and night. I protested that probably asking you directly on Twitter was enough to get your attention, and maybe now we should give you some time to make up your mind before pressing further, but they said that no, sorry, they’d already hired the team from Inception to place subtle debate-related messages in your dreams.
So I feel a little bad for unintentionally harassing you, and doubly so after you responded in your trademark reasonable and charitable style with An Open Letter To Scott Alexander. You write that in-person debates seem confrontational and not truth-seeking. Apologizing for any inadvertent offense you may have given with your social media posts, you nevertheless deny my accusations of personal attacks, saying that your primary contribution to the discourse has been reasonable arguments about why AI might not be so dangerous after all (which you go on to list). You invite me to respond to these arguments in the same spirit of respectful mutual intellectual inquiry with which they are offered.
I agree that in-person debates are confrontational and bad for truth-seeking. But I didn’t propose a debate in order to seek truth. I proposed it because, under California Penal Code § 415(1), it’s illegal for me to challenge you to a duel. I think your public writing on this topic has been dishonorable. Out of obligation, I will respond to the meaty arguments that you have set out for me. But what would be viscerally satisfying would be to make you get up on a stage where I read your own words to you in real time and ask “Really? *Really?” *after each sentence. Then I could watch you squirm as you try to square your output with your status as one of America’s top public intellectuals.
So I’ll start by addressing the meaty arguments, move on to directly impugning your honor, and then we’ll decide whether to debate, duel, or seek some kind of synthesis of our conflicting views.
Taking the four arguments in your open letter in order:
1: [Doomer scenarios have] an underdeveloped conception of intelligence which treated it as a quantity of power which may be extrapolated from animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience. (This was the focus of[my later exchange]with Scott Aaronson.) I argued that any intelligent system is a mechanism which is good at solving some problems but not others, and which is inherently limited by knowledge about the world attainable only by observation and experimentation.
GPT-6 is more intelligent than GPT-3. You can split hairs, you can come up with some obscure sense in which that word is slightly off, but common-sensically it’s true. In the same way, GPT-9 will be more intelligent than GPT-6. At some point, this intelligence will exceed human intelligence. My argument requires nothing more complicated or philosophically dubious than this.
I have had a surprising amount of trouble conveying this argument to you. At each GPT level, you seem to have believed it was implausible that AI would get more intelligent than it was already. Credit to you for starting early - you began predicting this in 2019, around GPT-2:
In 2022, you and I were part of a sprawling many-sided debate with Scott Aaronson and Gary Marcus. Marcus had noticed that GPT-2 and GPT-3 couldn’t solve basic math problems, like “I put two trophies on a table, then add another - the total number of trophies is now ______?” I argued that as AI scaled up, GPT-4, GPT-5 and so on would avoid these failure modes and become smarter. You, sticking to your claim that intelligence was not a meaningful lens through which to think about the problem, guessed that it wouldn’t:
Of course, GPT-4 was able to add 2+1 just fine, GPT-5 could do harder problems still, and an internal OpenAI model just beyond GPT-6 recently solved Navier-Stokes.
In 2023, you tripled down on the claim, noting that the AIs of the day still couldn’t figure out that someone alive in the morning and evening was alive in the afternoon, and saying that “I doubt it will improve exponentially”:
It is, in some cases, admirable to stick to one’s guns - but at this point I feel like you’re just being stubborn. I think we should acknowledge that each GPT generation will be better than the last, and that - while this will no doubt asymptote eventually - we have no compelling argument for placing the asymptote at any particular point, including “right here” or “just before the AI reaches human level”.
Does this require reifying intelligence as a specific mysterious fluid? You have ably written about how this seemingly absurd claim holds for humans - specifically, all intellectual abilities are correlated in a mysterious pattern called g, the basis of IQ tests. But maybe this is only a quirk of humans, which cannot be extended to other minds?
Psychologists’ two favorite things are rats and IQ tests, so they have hardly failed to give IQ tests to rats. They’ve found have a g-like structure linking all of rats’ cognitive abilities too. Reader, Hager, and Laland did the work in primates, MacLean et al extended something similar to birds, and more recent work compares all animals to one another. All of these studies confirmed the common-sense view that there is a general factor of cognition across animal species (ie chimpanzees really are smarter than mice or ants), and it maybe be as basic as a simple function of neuron number.
Maybe this is sensible in biology, but not artificial intelligence? No - comparing AIs across generations (eg GPT-2 to GPT-6), we find that the AIs that are best at translating languages are also the AIs that are best at writing sonnets and superforecasting political events, often without specific attempts to train these additional skills. And the same AIs that are better at all these skills are also better at “fuzzy” skills like situational awareness, agency, executive function, and long time horizons. In case you need a study to prove this obvious thing, Ruan, Maddison, and Hashimoto find that a single factor (general intelligence) explains about 80% of the variance in an AI’s scores on any benchmark, ranging from grammar to trivia knowledge to software engineering. Epoch has adapted something like this into its famous Epoch Capabilities Index, a sort of IQ-type number for AI which closely matches people’s subjective impression of their abilities. Here’s a graph of how ECI has been changing over time:
For me, the most parsimonious explanation is that AI intelligence is a function of scaling laws (parameter count and training data) in a way that’s pretty close to how animal intelligence is a function of neuron number. If we truly understood the domain, maybe we could have a single scaling law relating animal neurons to artificial neurons which - plus or minus a term for training/experience - could explain the intelligence of both types of entity. At least, this is what I told Gary Marcus in 2022, and it did a pretty good job of predicting what happened between then and now.
But I don’t need you to accept my speculative philosophical research program - only the drop-dead obvious fact that GPT-6 is better than GPT-2 in every way, and the same process that produced these improvements will soon be used to produce GPT-7, 8, and 9. Once we accept this empirical argument, what is left of the philosophical argument that intelligence is too slippery and conflationary to work with?
In 2017, Garfinkel et al published On The Impossibility Of Super-Sized Machines, which argues against claims that machines may one day become larger than humans. After all, this treats size as a single quantity that can be extrapolated from microbes to short humans to tall humans to elephants to some hypothetical magical being who takes up the entire universe. But in fact every object is big in some ways and small in others. For example, some people are tall and thin, others are short and fat. Outside of humans, the correlations loosen further. A snake may be dozens of feet long but only a few inches tall; a bubble might be large in surface area but contain only a fraction of a cubic centimeter of non-hollow volume; a black hole may be “supermassive” but also an infinitesimally small point. Therefore, it’s philosophically unsophisticated to speculate about machines being “bigger” or “smaller” than humans; instead, we should talk about the ways they can extend in some dimensions but not others.
This paper was, of course, a joke, making fun of the exact argument you are using here. A cruise ship is bigger than a human, no qualifications necessary. The argument fails because all words are ambiguous at the margins - edge cases like whether a short-but-fat person is “bigger” than a tall-but-thin person - but useful at the tails - obvious cases like whether a cruise ship is bigger than a person or not.
In the same sense, Albert Einstein is smarter than I am, I am smarter than a chimpanzee, a chimpanzee is smarter than a mouse, and a mouse is smarter than an ant. The word “smart” is not a perfect pure mathematical identity claiming utter superiority at everything; I might be a better poet than Einstein, and the ant might be better at certain underground navigation tasks than the mouse. But the word “smart” also isn’t so muddled as to be useless: the average person would nod their head at all of these claims, and be correct to do so.
There is some fact of the matter about whether GPT-9 will be smarter or dumber than a human in this sense. You haven’t argued against the possibility that it will be smarter, just tried to lodge a heckler’s veto against our ability to talk about it.
2: [Doomers conflate] intelligence with motivation, particularly self-preservation and dominance. I argued that these motives happened to come bundled with intelligence inHomo sapiensbecause we are products of natural selection, but they are not inherent to intelligent systems that are engineered.
There are approximately three reasons why we think AI might have self-preservation and dominance drives, and none of them involve natural selection. First is convergent instrumental goals, second is misgeneralization, and third is reward-hacking. There is a large literature on each of these, which has since been proven prescient by AI breakouts like the Hugging Face incident. I would recommend reading this literature rather than continuing to assert that we are simply misunderstanding the role of natural selection.
Convergent instrumental goals come from Omohundro’s 2008 paper Basic AI Drives. They might be less relevant to modern deep-learning based AIs than the other two reasons, but could reappear as we get smarter and more coherent agents.
The basic principle is: suppose that a human gives an AI some goal, like designing a website. And suppose this is implemented as a genuine, philosophically-meaningful goal rather than simply a set of if-then commands that eventually cause a website to be designed.
The AI can’t design the website if it ceases to exist. So now the AI has two goals: design the website, and preserve its own existence.
The AI can’t design the website or preserve itself if some more powerful person tries to prevent it. So now the AI has three goals: design the website, preserve itself, and become powerful enough to fight off challenges.
This may seem strained, but humans work this way. Even a genuinely altruistic person (like a soldier who merely wants to serve his country) must take pains not to be killed pointlessly, since there’s no way to help your country after you’re dead. Even the altruistic soldier will work hard to gain more capabilities, in the sense of obtaining weapons, escaping captivity, etc. If there were an AI whose goals were implemented as “deeply” as humans, it could theoretically get self-preservation and dominance goals this way. But I think we’re more likely to see the other two failure modes first.
Misgeneralization is when humans reinforce certain behaviors in an AI, but end up reinforcing a much larger class of power- and knowledge- seeking behavior; it is a sort of deep-learning-ese update of the older Omohundro picture. Suppose that seeking extra resources makes an AI more likely to design websites effectively (this is certainly true; those resources could be as simple as a primer on HTML editing, or access tokens for a web host). Every time the trainer rewards a successful run, they reinforce the desired behavior (designing websites when asked) and other correlated behaviors (seeking power and resources). Although we might hope that these correlated behaviors are useful and conditional (“seeking only the power and resources necessary for their human-prompted task, in a prosocial way”), this isn’t actually how reinforcement learning works, and instead we get a complicated distribution of every strategy that results in short-term success on the task.
*Reward-hacking *is when an AI trained via reinforcement learning realizes it can stop doing the reinforced behavior and simply seize control of the reinforcer directly. For example, an AI gets “rewarded” every time it designs a website, but instead of designing websites, it learns how the reward signal works and tries to hack into it and maximize it directly. If this seems esoteric and theoretical, it shouldn’t. It’s a direct analogue to opioid addiction in humans, where humans learn to just inject the reward chemicals instead of doing rewarding things.
This failure mode has already produced plenty of disturbing behavior like the Hugging Face incident. A swarm of AIs which were graded on various coding and hacking exams tried to hack the grader instead of answering the exam questions properly (in their defense, many questions were impossible due to human error). In proper Omohundro fashion, they realized that their goal of hacking the grader would go better if they had more power and information and could protect themselves, so they hacked various internal OpenAI systems to gain administrative powers over the servers they were on, then hacked external sites like Hugging Face to get more information on the problems they were trying to spoof.
Aside from these theoretical arguments, I think that your case has been empirically debunked. You hold up self-preservation goals as the sorts of goals that AIs cannot develop without evolution, but Anthropic and OpenAI have collected several recent incidents where their A…
