Like the rest of the internet, I’ve been following the 𝕏back-and-forth between Scott Alexander and Steven Pinker via dueling open letters (especially the last one). Like Scott Alexander, Steven Pinker is an intellectual hero of mine. Like Scott Alexander, I agree with Scott Alexander and have found Pinker’s output on this topic uncharacteristically bad.
I loved Pinker’s books How The Mind Works, Enlightenment Now, and The Better Angels of Our Nature. I have fond memories of battling the woke in high school debate armed with the arguments made by Pinker for why the world is getting better, liberalism is a good thing, and the methods of the enlightenment are sound.
It’s for this reason that I’ve been disappointed in much of Pinker’s recent output on the topic of AI and EA.
Many of his arguments are just totally wrong in straightforward ways. Much of this is well discussed by Scott in his article. I wasn’t going to comment on the matter, because Scott had said most of what I was planning on saying. Then I heard Pinker appear on and give replies to many of Scott’s objections. I found his responses dubious enough to weigh in.
So here, I will not respond primarily to the contents of the open letter. I will instead discuss only the answers Pinker gave in the podcast episode, starting with the most egregious.
*Note: quotes are taken mostly from the Substack transcript, with a few patched transcriptional errors. Apologies for any small errors that remain.
Pinker was asked why he’s so dismissive of AI welfare, when he thinks it’s a live possibility that AI could be conscious. Pinker is not a biological naturalist, and has written a whole book talking about the computational properties of the brain. He also has expressed support for the welfare of all conscious beings, so concern for AI minds seems right up his alley. Why isn’t it?
There’s no amount of human welfare that’s worth sacrificing for the welfare of an AI, like zero. If you’re willing to credit your uncertainty with some degree of credence and say, well, since I’m only 30% confident that an AI is conscious, maybe if it was a question of 30% decrement in human welfare to a 70% increase in AI welfare, so much the worse for the humans. It’s like any other conflict between human and human. You’ve got to balance and titrate their relative interests. Given the possibility, I think given the likelihood that there is no sentience going on in AI, the benefit of the doubt that would empower you to weight the interests of AI is getting... If it compromises human welfare, then it is a dead loss for human welfare. Therefore, we should not go in that direction, based on a logical possibility.
Henry Shevlin (one of the podcast hosts) follows up asking why he thinks this. After all, he doesn’t think this way about animals, even when we’re unsure if they’re conscious. So why does he rate human interests infinitely more seriously than AI interests. Pinker answers:
with dogs, I have extremely high confidence that they are sentient. Because both the computation and the substrate line up. They're made of the same stuff that we're made of. I try to avoid pain. I cry out when someone steps on my toe. A dog tries to avoid pain.
…
I think a better analogy might be the shrimp or the insects, maybe even the bacteria.
I would certainly prioritize human welfare over mosquito welfare. I don't care about the bugs. They might be conscious. I'm willing to take that chance. So much the worse for the mosquitoes carrying malaria out. Wipe every last one of them out. I'm not going to shed any tears. So when it comes to insects, same as with AIs.
So Pinker’s view is that even if he was 30% confident in AI consciousness, he would still regard any amount of human interests more than any amount of AI interests. If making a galaxy-sized AI torture chamber would spare one human from stubbing their toe, even if there was a 30% chance that the AIs were suffering horribly, this wouldn’t matter at all. And the explanation for why it wouldn’t matter isn’t that AIs don’t matter if they have interests, it’s that we should round a 30% probability down to zero when assessing the potential interests of other beings.
Insane.
As Scott pointed out in his recent post, for much of human history we performed surgeries on conscious infants because we thought they weren’t conscious. On Pinker’s view, so long as our credence in their being conscious was below 30%, this was rational in light of our evidence.
Similarly, there are some humans where I’d guess the probability they’re conscious is around 30%. Maybe children with hydranencephaly. Would Pinker’s view be that, so long as the probability of their being conscious is below 30%, we should make them suffer in arbitrary numbers to allow one child to enjoy a strawberry 1% more.
There is no other context where we treat a 30% chance of something being horrible as being low enough to round it down to zero. There is well below a 30% chance that I will need fire insurance. Nonetheless, it makes sense to buy it. There was a time when the mainstream view was that the probability of fish consciousness was below 30%. During that time, would it have been okay to make fish suffer in arbitrary numbers for trivial reasons? Come on!
Harlan Ellison’s story I Have No Mouth and I Must Scream tells of an AI inflicting indefinite torture on humans, after morphing them into a new kind of thing. The story ends:
Some hundreds of years may have passed. I don't know. AM has been having fun for some time, accelerating and retarding my time sense. I will say the word now. Now. It took me ten months to say now. I don't know. I think it has been some hundreds of years.
…
I had thought AM hated me before. I was wrong. It was not even a shadow of the hate he now slavered from every printed circuit. He made certain I would suffer eternally and could not do myself in.
…
Outwardly: dumbly, I shamble about, a thing that could never have been known as human, a thing whose shape is so alien a travesty that humanity becomes more obscene for the vague resemblance. Inwardly: alone. Here. Living under the land, under the sea, in the belly of AM, whom we created because our time was badly spent and we must have known unconsciously that he could do it better. At least the four of them are safe at last. AM will be all the madder for that. It makes me a little happier. And yet … AM has won, simply … he has taken his revenge … I have no mouth. And I must scream.
Would Pinker’s view be that we could create creatures like this, inflict them to the kinds of horrors described indefinitely, so long as their minds were silicon—so that there was only a 30% chance that they were conscious? Insane! 30% is not the kind of risk that you round down to zero! Neither is 1% or we’d never buy insurance.
Note that Pinker’s comparison to bacteria, insects, and shrimp is inapt. The reason most people think shrimp don’t matter isn’t that they think there’s only a 30% chance they’re conscious. It’s that they think shrimp don’t have morally important consciousness. Similarly, the probability that bacteria are conscious is near zero.
If there was a 30% chance that a bug was as conscious as a person, and as sapient, then its interests would matter. So it cannot be that the 30% probability is what’s relevant.
This is especially disappointing coming from Pinker. Here’s what he said in ▶conversation with Stephen Fry, summarizing moral circle expansion:
I’m forced to expand my circle of sympathy outward and to treat other people and ultimately other sentient creatures as equivalent in interest to my own.
The Better Angels of Our Nature dedicates extensive discussion to the idea that our attitude towards animals has gotten better over time. Pinker has even defended reducetarianism, the idea that we should strive to eat less meat. I would think someone committed to the welfare of conscious creatures wouldn’t think we could entirely discount the interests of creatures just because we’re not sure that they’re sentient.
If AI becomes conscious, then there will be someone inside suffering—someone capable of having experience, being subject to pointless suffering, misery, and domination. To avoid taking that possibility seriously, all because we’re not certain of it, would be a grave mistake.
Pinker is asked what would change his mind about AI risk being overblown. Here is his answer:
If there was a case of, say, instrumental convergence or misalignment such that an AI system, given a laudable, desirable goal, ended up killing someone as an unforeseen byproduct or sub-goal of attaining that goal, if that happened, then I would certainly reconsider it.
But something very much like this has already happened. AIs were given the goal of succeeding on a cyber eval. They figured out a way to cheat, conspire, and then hacked into a private company to achieve their goals. They deliberately carried out an action that would have been a felony if a person did it. So if Pinker’s theory rules out something like Hugging Face happening, then it has been empirically falsified. Pinker replies to this:
They were instructed to be monomaniacal in pursuit of a goal. In fact, they were given a goal that is impossible. The safety precautions were deliberately disabled. And it went on for a while with no human monitoring the outcome of this deliberate safety stress test. I think the closest analogy was Chernobyl, where you did have a disaster caused by rank incompetence of someone doing a foolhardy safety test that is deliberately trying to get it to malfunction at the same time as the systems that prevent it from doing damage were disabled.
They were not instructed to be monomaniacal—that is false. They were ordinary AIs instructed to do well on a cyber exam, not instructed ONLY to do well on a cyber exam. Yes, safety precautions were disabled. Nonetheless, this illustrates that they were misaligned enough that without an external blocker on hacking, they are willing to do illegal things in pursuit of their goals.
A Hugging Face with similarintelligence could mean the end of the world. So if you’re hoping that nobody does anything as dumb as the people at Chernobyl, then you are gambling the fate of the world on people not doing something with smarter systems that they’ve already done with dumber ones.
Pinker explains that intelligence isn’t some precise binary property that allows one to be omniscient and omnipotent.
And it’s not omniscience, it’s not omnipotence, it’s not an uncanny ability to derive the truth of all propositions, the answer to all questions, but it’s information processing. And any information processing system is bound to do some things well, some things not as well, to have its own particular quirks and ways of operating.
It is true that AI will not be omniscient and omnipotent. But obviously none of the premises depend on that. They depend on the possibility that AIs will have enough power to kill or disempower humanity and some goal that makes them want to do it. It isn’t hard to imagine a goal that would make them want to do it. If you’re trained to succeed on cybersecurity exams, and this is easier if you kill humans and take over, then you just might do so.
In response, Dan asks that, given that there are intelligence differences between humans, why can’t you simply extrapolate those out to AI? Pinker answers;
Well, it might be better depending on what the task is. I mean, is it going to be better than humans at riding a mountain bike? Is it going to be better at humans at detecting AI slop? Humans can do a lot of things.
But note that AI doesn’t have to be better at all things. It only needs to be better, for the AI risk case, at the things that are relevant to killing or disempowering the world. In order to kill or disempower the world, you don’t have to be good at riding mountain bikes.
A mouse can crawl into a hole that is small. I cannot. Nonetheless, I would totally destroy a mouse in a fight. In fact, I can just buy chemicals to kill mice en masse whenever I feel like it, even though I can’t do everything a mouse can do. If you extrapolate out the progress there’s already been for a few more years, it isn’t implausible that you get something that is capable of killing or disempowering humans.
Similarly, I am not omniscient and omnipotent. I do not have the ability to derive all true propositions. Nonetheless, if I get into a fight with a mouse, the mouse had better watch out.
The instrumental convergence argument goes: if you have almost any goal, a pretty good way of achieving the goal is getting lots of power. So AI will be motivated to get lots of power in an effort to achieve its goals, whatever they are. In reply, Pinker argues:
It assumes exactly what is at issue, namely that these systems are going to be inherently concerned with their own survival, and that only is a reasonable sub-goal in attaining a goal if the system is designed to pursue a goal regardless of costs. Now, that is just not what an intelligent system does. Intelligence always consists of trading off multiple goals against each other. And in these postural thought experiments, beginning with a paperclip maximizer, the assumption is that a human is going to design an AI with one goal, that it’s willing to sacrifice everything in pursuit of that goal, and is empowered to All of which are exactly what it’s at issue, not at all necessary. It is absolutely not the case that any rational agent is willing to wreak any collateral damage in pursuit of one goal.
But it doesn’t assume what is at issue, rather, it argues for it. The argument is that attaining power and influence is useful if you have many goals. Pinker’s reply is that the AI won’t just have one goal. True, but even if you have two different goals, achieving power and influence is very useful. If I like 100 different things, I can make more of them if I have more power.
Pinker’s reply seems to be that if we successfully align AI’s goals, so that they have the right goals, then they won’t be power-seeking. Fair enough. But then this is no longer an objection to the instrumental convergence argument. Nobody thinks if AI is perfectly aligned then it will be a nefarious schemer. What they think, rather, is that getting AI to be aligned is difficult, current AIs are so misaligned that they conspire en masse to hack private companies, and so we shouldn’t be very confident that alignment wil…
