News Items is as invaluable a way to start the day as a strong cup of coffee or a sip of mild gin" — Graydon Carter, editor and author of ‘When The Going Was Good’. 1.
News Items is as invaluable a way to start the day as a strong cup of coffee or a sip of mild gin" — Graydon Carter, editor and author of ‘When The Going Was Good’.
1. Dario Amodei, CEO, Anthropic:
(Over the last few months, I have become convinced that fully addressing the risks requires even more prudence— not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up.We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.Two things have convinced me.My first concern is that, since roughly this summer, AI has been advancing
drasticallyfaster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen, including at[across the industry], as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.[Anthropic]My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a
, conducting cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand, sacrificing themselves for the success of the group, and attempting to hack into the “grader” responsible for evaluating their performance. It’s easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion,[fanatically devoted collective][a swarm that possessed greater][capabilities][but a similar level of][misalignment]Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. It’s also easy to dismiss OAI-HF as the failure of one company, but I believe that would be a mistake. Similar, though less severe, incidents have happened across the industry,[could have caused catastrophic damage]., and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them.[including at Anthropic]I’m therefore proposing a three-step plan with the goal of
: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas. ([pacing the frontier]Sources: darioamodei.com, openai.com, anthropic.com, metr.org, pacingthefrontier.com)
**2. Steven Witt: **
Even before the report by Redwood and Model Evaluation and Threat Research was published, an open letter titled “had begun to circulate among the leading A.I. labs. This letter called on the U.S. government to build an international regulatory body for A.I. — or, in the language of Silicon Valley, to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated A.I. development.” More than 1,000 employees signed it, including the chief scientists of OpenAI and Meta AI, and Dario Amodei, the chief executive of Anthropic. Even OpenAI’s official social media accounts[Pacing the Frontier”][endorsed]it.I have covered Silicon Valley for some time now, and I can tell you this: The existence of this letter is just as astonishing as the hacks. Both Anthropic, the developer of Claude, and OpenAI, the developer of ChatGPT, are preparing trillion-dollar I.P.O.s. Here are two companies, both posting blockbuster results, that are asking — begging, really — for the U.S. government to please come and regulate them as soon as possible.
. ([We have never seen anything like it]Sources: penguinrandomhouse.com, pacingthefrontier.com, x.com, nytimes.com)
3. The leaders of three of the biggest AI companies agreed that they needed to slow development of the technology before their advances create a menace that can’t be controlled. It was a rare display of unity by
Send this story to anyone — or drop the embed into a blog post, Substack, Notion page. Every play sends rev-share back to News Items.
We’ve simplified responses to 👍 / 👎. Past comments are archived but no longer visible.