It is maddening how often people are extremely confident in views that aren’t supported by any very good arguments. To rationally be extremely confident in some view, you need an extremely good reason to believe it.
For example, the proponents of biological naturalism—the view that biology is needed for consciousness—tend to be almost certain the view is correct (or act like they’re almost certain). Their credence generally appears to be well over 90%. Yet what licenses such confidence?
It is usually unclear. There are few agreed-upon arguments for biological naturalism. The arguments given in favor of biological naturalism vary from biological naturalist to biological naturalist and tend to be very weak. I can understand thinking odds are decent that biology is needed for consciousness. I don’t agree with that view, but I at least understand it. But I don’t get being almost certain of the view. There is no single canonical obviously-correct argument for it (note: naming some random unique feature of biology isn’t a very strong argument, because we don’t know that the unique feature is needed for consciousness).
There are lots of disputes that aren’t settled by the evidence. In the case of consciousness, we really don’t know what ingredients are needed for consciousness. So we should have uncertainty about whether they require biology and whether they’d arise naturally in LLMs. And yet I find it very common, in these disputes that are underdetermined by the evidence, for people to be extremely confident, often for reasons that seem very bizarre to me.
You might object that we can be confident that some feature is insufficient for consciousness without knowing what features are needed for consciousness. I’m pretty sure rocks aren’t conscious, even though I don’t know what is needed for consciousness. This is correct, but it’s totally unclear why people are confident that what LLMs do is in principle incapable of giving rise to consciousness.
I can explain why I think it’s unlikely rocks are conscious (they perform no computational functions, have none of the neural correlates of consciousness on leading theories, don’t do anything like making decisions, have no intelligence, have no internal representations, etc). But what feature will LLMs definitely indefinitely lack that you’re super confident is needed for consciousness? Autopoiesis? Why? Why are you confident that’s needed for consciousness? I don’t get it.
Here’s another example: lots of people are very confident that the AI industry is a bubble that will pop soon. Many of these people have been predicting for a long time that AI would plateau. Now, it’s *possible *that progress stops soon. Certainly progress will *eventually *stop. But what in the world licenses extreme confidence that it will stop before we get superintelligence?
I think the rational credence to have in that is below half, because it looks like we’ll get superintelligence soon on the current trajectory. If progress has continued for years, even speeding up, you should think it probably won’t stop in the next year. But at the very least, it seems totally irrational to be near certain that it will stop. The Ed Zitrons of the world who repeatedly predicted LLMs plateauing with high confidence weren’t just wrong—they were being irrational. They should not have been as confident as they were.
Now that I think about it, I consider this to be a pretty core part of my worldview: don’t be super confident that something’s true unless there is a very strong argument for thinking it is. Your confidence in a proposition should be proportional to the arguments in favor of it. I think almost everybody violates this rule.
One case where I violated this maxim was in estimating AI risk. For a long time, my p(doom) was around 2%. Now it’s nearer to 12%. I think my credence of 2% was irrational. I should have known better. I should have been less confident in non-obvious things.
We get doom if the following things happen:
We continue along the current trajectory, which is capable of reaching superintelligence.
The superintelligence we build is misaligned.
We don’t get a decisive warning shot that leads us to shut it down.
A misaligned superintelligence can kill or disempower the world.
I thought there was about a 90% chance that we’d solve alignment. In hindsight, that seems way too high. It’s not at all obvious that if there’s a race between companies trying to build superintelligence as quickly as possible, they’ll be successful in making it aligned in time. A big part of my reasoning was that it seemed like the models at the time were aligned—but I don’t think that provides strong evidence that future models would be aligned (see Hugging Face). There is no extremely strong argument for thinking alignment will work; for this reason, it was irrational to be so confident.
The other steps mostly seem pretty plausible too. Thus, there are a number of things that are each plausible. If they all happen, everyone dies or is permanently disempowered. This is cause for extreme alarm; it would be irrational to be very confident that things will go alright. If you are very confident that things will go okay, then think about which step you’re very confident won’t happen. Are you confident that we won’t build superintelligence? Are you confident that the world can survive building superintelligence?
Lots of AI skeptics seem to be confident that even a misaligned superintelligence wouldn’t kill everyone. I think this is irrational. It’s hard to know what a superintelligent being is capable of. Because it’s superintelligent, we just don’t know what it will do. Are you really confident that a thing that surpasses us across the board as thoroughly as Stockfish outclasses Magnus Carlsen won’t be able to figure out a way to kill or disempower humans? It can’t think of any way—canvassing its knowledge of cybersecurity, biology, nanotechnology, and a hundred other areas—to kill or disempower humans? You’re near-certain of that? Really?
I feel similarly about the MIRI-type people who are almost certain that AI will kill everyone. I realize they give arguments for this, but I don’t think it would be rational to be extremely confident that we won’t get a decisive warning shot, say, or that alignment won’t work. For this reason, p(doom)s much above 70% seem irrational to me.
Often, people act super confident in views without really giving them much thought. Almost everybody is very confident that meat eating is fine before they think seriously about it. But this seems like a mistake. If you don’t know the conditions that animals live in and you haven’t thought about animals’ moral status, you shouldn’t be confident that it’s okay to eat them. Killing another sentient being is, whatever you think about its ultimate permissibility, a grave matter: it isn’t the kind of thing that you should at the outset be near certain is okay.
Overconfidence without decisive arguments is especially common when thinking about simple animals like shrimp and insects. It’s popular to count these organisms’ moral status by their neurons. So, because a honeybee has about a million neurons, and humans have about 86 billion, this view holds that our conscious experience is in expectation 86,000 times more intense than that of a honeybee.
But note: if you think that, then you think that there isn’t even a 1% chance that honeybees have experiences 1% as intense as humans. You are near certain that your experience of eating porridge or dreaming is vastly more intense than a bee’s experience of having sex, starving to death, or being burned to death. You are near certain that inside the bee, while maybe the lights aren’t totally off, they are turned down to such a degree that there’s almost nothing of note happening.
But why would you think that? Why are you extremely confident of that? Sure, maybe it’s right. But what makes you more than 99% confident that it’s true? I’m not even 99% sure that degrees of consciousness make sense, much less that bees have experiences no more than 1% as intense as human experience.
There are some biological features that can be implemented robustly in small organisms. Small organisms can have pretty robust eyesight and functional wings. It’s not like a bee can only fly one 10,000th as well as a bird. It’s unclear why we’d be near certain that intense pain requires a large brain. This is especially true given that small animals behave like they’re in a lot of pain.
Drop a crab into a pot of boiling water and it will behave more like if someone dropped *you *into a pot of boiling water than if someone stubbed your toe. So a not-too-unlikely inference is: maybe when you drop them in boiling water, they feel sort of like you would if you were dropped into boiling water. If you’re super confident that’s false, why?
This is also an important lesson to take to heart when you’re an outsider evaluating a disagreement between experts. Suppose that there’s a big disagreement between two economists. They have a long back-and-forth. I can’t really follow every detail super well. Either one would crush me in a debate if I tried to defend the other side.
In my view, it would be irrational to be supremely confident that one side is right and the other side is wrong. Even if one side sounded a bit more convincing to you, that doesn’t give you super strong reason to think they’re right. I often encounter lay people who have confident views on random disputes between experts—I always find this baffling. What makes them so confident?
Let me address one possible misunderstanding of what I’m saying. I’m not suggesting that there needs to be strong evidence against a view in order for you to be confident that it’s false. There are some things that are very likely false, not because there’s good evidence against them, but because they’re intrinsically improbable. For example, I have no specific evidence that I don’t have 123,387 hairs on my head, but I think it’s unlikely, because it has a low prior probability. It’s unlikely before we consider the evidence.
So I’m not suggesting you need strong positive evidence against a view to be confident that it’s false. I’m suggesting you need a good reason to think something is false. That reason can be “it has a very low prior probability.” But then there ought to be some convincing reason to think its prior is low. My sense is a lot of the biological naturalists treat non-biologically-naturalist theories as having a very low prior. But I’m not quite sure why they think this.
People most commonly write off plausible views when they would have worldview-upending implications. If there’s going to be superintelligence around the corner, that changes everything. If insects can suffer intensely, so that almost all the intense suffering in the world is experienced by arthropods, that radically upends our moral priorities. Can we be rational to change our lives based on a conclusion without super strong evidence for it?
Yes. It is irrational to have your assessment of the probability of a conclusion depend on the implications that its being true would have. Your bar for concluding that an asteroid is likely to be coming to Earth shouldn’t depend on how devastating it would be if it did. That makes no difference to the probability it will hit Earth. Your credence that arthropods suffer intensely shouldn’t depend on the moral implications that would follow if they could suffer intensely—it isn’t as if learning that the world had few arthropods would make it likelier they could suffer intensely, by making the conclusion lower-stakes.
There are some conclusions that you should take very seriously even if you’re not sure that they’re right. Some possibilities are significant enough that even a 30% chance of them is a big deal. Sometimes, there’s a 30% chance that something is true because it has a not-very-low prior and there’s no good evidence against it. In such cases, you should change your priorities based on a conclusion, even without any very strong reason to think it’s true, because it’s plausible and huge if true.
High confidence requires good reasons. That a conclusion would be shocking or inconvenient if true on its own gives you no reason to believe that it isn’t. Overall, uncertain and possible worldview-upending conclusions are highly underrated—we ought to take them much more seriously.
