AI companies like Anthropic and OpenAI have long mentioned the possibility that their products could be existentially dangerous, but this has recently crescendoed into even more dire warnings, including suggestions that we have to pace the frontier, ie coordinate to slow AI progress.
In response, cynical commentators have suggested this is a conspiracy to boost the companies’ stocks. The exact motives imputed vary from person to person. Some say the companies are hyping the product by talking about how dangerous it is; if AI is smart enough to kill all humans, it must be smart enough for your business workflow. Others say they’re trying to distract from near-term harms; every inch of newspaper space devoted to human extinction is one less inch that can be spent on plagiarism and water use. But the most popular story involves regulatory capture: if the product is unsafe, maybe it can only be trusted to a few big politically-connected companies, and everyone else should be banned.
Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb.
Taking these in order:
Grace et al surveyed 1,580 AI researchers who presented papers at big AI-related conferences, mostly academics with no company affiliation. Over half thought there was at least a 10% chance AI would cause “human extinction or similarly permanent and severe disempowerment of the human species”. 72% said we needed to be prioritizing AI safety research more. This survey was in 2024; current numbers would probably be higher.
Geoffrey Hinton, Yoshua Bengio, and Yann LeCunn are collectively considered “the godfathers of AI” for inventing deep learning; in 2018, they received the Turing Award, sometimes called the Nobel Prize of computer science, for their accomplishment. In 2023, Geoffrey Hinton quit his job at Google to work full-time raising awareness of the potential existential dangers of AI; he said there was “between 10% and 50% chance” that his invention would destroy humanity. In the same year, Yoshua Bengio also switched to activism, saying that he “regretted his life’s work” and that AI could have a “catastrophic impact”. Neither was affiliated with a major AI company.
(the third godfather, Yann LeCunn, runs a $3 billion AI company, and says he isn’t worried)
Along with these independent researchers, there are many others who formerly worked at companies, then quit to whistleblow about the dangers, or left to pursue anti-AI activism full-time. Paul Christiano left OpenAI in 2021 to found an AI alignment nonprofit; he later said there was a “50/50 chance of doom shortly after you have AI systems that are human level.” Daniel Kokotajlo quit OpenAI in 2024, refusing to sign a non-disparagement clause even though it could have cost him $2 million in equity, in order to start a nonprofit warning about risks from AI (he says he thinks there’s about an 80% chance AI kills everyone). Jacob Coxon quit Anthropic in September, also likely giving up millions of dollars in equity, because he thought it was “gambling with our lives”. There are dozens of others.
The people at AI companies are no more likely to raise the alarm about AI dangers than people who were never at AI companies, or people who were previously at AI companies but have since quit.
**DARIO AMODEI **first started discussing the risk of superintelligence destroying humanity in 2013, before OpenAI or Anthropic was founded, and when Amodei wasn’t employed in tech at all. Here’s a 62 page debate between Amodei, Eliezer Yudkowsky, and other AI safety enthusiasts about strategy, dated October 27, 2013:
In 2017, when Amodei was an ordinary employee at OpenAI, he spoke about his work the on the podcast 80,000 Hours (July 21, 2017):
I mainly work on safety. We have a safety team that’s so far, myself and Paul Christiano. Paul was co-author of mine on the
[[Concrete Problems In AI Safety]]paper and has also written a lot online on his blog about AI. He’s probably one of the people who’s I think done the most to promote clear thinking about the problem and tie it to current AI. We have a third person joining in a few weeks who I’m super excited about. We’re trying to build up a team that focuses on technical safety. We also do a little bit of strategy stuff which is how do we get different organizations that are working on AI to cooperate with each other, how do we cooperate with policy makers on questions like these. We’re also thinking a little bit about those issues but mainly technical safety.
Dario expanded on the cooperation point - he told the host about his fear that AI would lead to an “arms race” in which companies competed to develop the technology first and let safety slide. His solution was making companies cooperate:
We all think about AGI and think, that safety issues are important. . . then the question is, how does that scale to there being a lot of organizations? How does that scale to others who get involved once they see how powerful AI is? Can we make them cooperate as well? My hope is that we can.
This became an obsession with him, and he hired an advisor to help OpenAI work on the problem. According to the New Yorker:
In 2017, Amodei hired Page Hedley, a former public-interest lawyer, to be OpenAI’s policy and ethics adviser. In an early PowerPoint presentation to executives, Hedley outlined
how OpenAI might avert a “catastrophic” arms race—perhaps by building a coalition of A.I. labs that would eventually coördinate with an international bodyakin toNATO, to insure that the technology was deployed safely.As Hedley recalled it, [OpenAI president Greg] Brockman didn’t understand how this would help the company beat its competitors. “No matter what I said,” Hedley told us, “Greg kept going back to ‘So how do we raise more money? How do we win?’
This incident and several like it turned Dario against Greg, leading to an era of constant backstabbing office politics. Dario led a pro-safety faction against Greg’s pro-business faction; Greg usually won. It didn’t help that Dario was anxious to the point of paranoia, constantly insisting that their work was too dangerous to release:
Sam Altman tried to play peacemaker by promising Dario various concessions, but eventually Dario noticed that Altman lied constantly and never came through on any of them, so he and the rest of the safety faction left to found Anthropic. Here’s how Fortune described the split:
Amodei, along with a small group of employees, became concerned about OpenAI’s direction and what he perceived as a lack of safety involved in scaling up ChatGPT.
In 2021, Amodei, along with his sister and cofounder Daniela Amodei and a dozen other ex-OpenAI employees started Anthropic with a mission to put safety first. The company released its large language model, Claude, in 2023.
The employees who jumped ship from OpenAI believed two things, Amodei told
Fortune’s Jeremy Kahn at theFortune Brainstorm Techconference in 2023. First, they believed if you pour more compute into large language models, they will greatly improve, and there’s almost no limit to how advanced they can get. Second, they believed—in addition to scaling up the models—there needed to be a strong set of guardrails and safety considerations.
SAM ALTMAN said in February 2015 - that is, ten months before he started OpenAI - that “development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” He continued:
The US government, and all other governments, should regulate the development of SMI [superhuman machine intelligence]. In an ideal world, regulation would slow down the bad guys and speed up the good guys—it seems like what happens with the first SMI to be developed will be very important.
Although my general belief is that technology is often over-regulated, I think some regulation is a good thing, and I’d hate to live in a world with no regulation at all. And I think it’s definitely a good thing when the survival of humanity is in question. (Incidentally, there is precedent for classification of privately-developed knowledge when it carries mass risk to human life. SILEX is perhaps the best-known example.)
He also famously said in June of that year that “AI will most likely lead to the end of the world, but in the meantime there will be great companies”. Some people say this was a “joke”. I think it was the one time in his life that Sam Altman was ever completely 100% honest.
**ELON MUSK, **another AI company leader who has been talking about extinction risk, has an equally long relationship with the topic. His first statement on AI safety was in 2014, before he had any business ventures in AI, when he said that “We are summoning the demon with artificial intelligence”. He continued by calling it “our biggest existential threat”, and admitted that “I’m increasingly inclined to think there should be some regulatory oversight, maybe at the national and international level just to make sure that we don’t do something very foolish”.
Here’s a photo from the first big AI Safety conference in Puerto Rico. Elon is circled in orange. He was a rare famous billionaire who agreed to hang out with powerless nerds to debate the safety of a technology that technically didn’t exist yet, just for the love of the game.
After the conference, Elon agreed to donate $10 million to the fledgling cause. He continued to be a regular at AI safety conferences over the next few years. I saw him speak at the Asilomar conference in 2017, where he gave a rambling talk on how brain-computer interfaces could solve the problem (they didn’t). ** **
**SHANE LEGG **and **DEMIS HASSABIS **are two of the three co-founders of DeepMind, now Google DeepMind. In 2008 - a year before founding Deep Mind - Legg wrote his PhD thesis on “Machine Super Intelligence”, where he wrote that it could be “a nightmare scenario” (although he also said that “if we carefully prepare for this possibility in advance, not only might we avert disaster, we might bring about an age of prosperity unlike anything seen before.”) His other optimistic predictions include “If by suffering you mean prolonged suffering, then I think this is quite unlikely. If a super intelligent machine (or any kind of super intelligent agent) decided to get rid of us, I think it would do so pretty efficiently.”
Hassabis was also concerned at least as early as 2012, when he reportedly tried to convince Elon Musk that AI was an existential risk. He attended the same 2015 AI safety conference as Musk (see photo above) and included as a condition of DeepMind’s acquisition by Google that Google set up a powerful AI safety oversight board (this went wrong in approximately every way possible).
THE CORPORATE PLOT hypothesis requires these CEOs to be the most prescient people in history. In 2008 or 2013 or 2015, when they were all in other industries and AI was just an academic curiosity, they would have had to realize that one day AI would become a trillion dollar business, that they would be the CEO of one of the companies, and that the company would benefit from regulatory capture. Then they would have had to devote a significant proportion of their life, and in some cases donate millions of dollars, to start laying the groundwork ahead of time, just in case it became necessary. Dario Amodei, while finishing his biology postdoc, would have to have moonlighted as an AI safety enthusiast just in case he became a multi-billion dollar tech CEO one day. Shane Legg would have had to choose his PhD thesis topic based on the possibility that he would own shares in a Google AI subsidiary sixteen years later.
The alternative is that all these people believed it then, and still believe it now, for the same reason that all of the independent experts and academics believe it. They are working in the field partly because they think that being closer to the development of AI gives them more of a chance to control it and keep it safe than if they were sitting on the sidelines.
Having shown that these men would have to be the smartest people in the world to pull this off, it’s worth noting that the plot they apparently dedicated their lives to is really dumb.
For example, if the Pacing The Frontier consortium attempt was a plot to pump stock prices, you might expect that, upon it being announced, stock prices would go up. This didn’t happen.
Anthropic and OpenAI aren’t publicly traded companies, so they don’t have an easily-tracked stock price. But there are complicated cryptocurrency bets called “perpetual futures” that traders use to approximate the stock prices of private companies (usually these are some kind of prediction-market like instrument on eventual IPO price). When the slowdown call happened, Anthropic’s value on the biggest cryptocurrency futures platform dropped 4.57%, and OpenAI’s stock dropped 13.42%.
(two other AI companies that endorsed the slowdown proposal, Google DeepMind and X.AI, are subsidiaries of larger publicly traded companies; Google and SpaceX respectively. Both companies went down modestly the day after the announcement, although they have so many other things going on that I take this less seriously than the AI company futures.)
Why wouldn’t the stock market (or at least this particular cryptocurrency simulation of the stock market) approve of the slowdown as a potential regulatory moat? It seems like traders thought the first-level effects (slowing down) were more important than any secondary regulatory effects that might or might not materialize.
Libertarian economist Alex Tabarrok is head of the Center for Study of Public Choice, the subfield of economics that studies things like regulatory capture. He agrees wi…
