The Scariest Recent AI Development
There was a recent AI development that has gotten basically no news coverage but that scared all the people I most respect in AI. In many cases, it scared them more than Hugging Face.

There was a recent AI development that has gotten basically no news coverage but that scared all the people I most respect in AI. In many cases, it scared them more than Hugging Face.
There was a recent AI development that has gotten basically no news coverage but that scared all the people I most respect in AI. In many cases, it scared them more than Hugging Face.
A recent METR report uncovered that AI progress at Anthropic has been going, in the last twelve months, about twice as quickly as it had in the previous year. The majority of this acceleration, around 80%, is due to AI being used to speed up AI research. This means that AI is massively accelerating progress in AI. AGI may be right around the corner—I’d probably put the odds of AGI in the next two years a bit above half.
This is very scary. One reason is that it just means things are going extremely quickly. But the bigger reason is that recursive self-improvement means that progress will make future progress even faster. As AI gets better, it gets better at improving AI research. Ajeya Cotra who works for METR and is one of the leading experts on AI risk in the world, wrote:
This is a much bigger deal than it might sound at first. I think this report is the most important update we have gotten about AI risk in the last couple of years — much bigger than
[the Hugging Face attack]. Naively, it points toward much shorter timelines than most external researchers (including[the AI 2027 team]) have projected.
Cotra came third in an AI forecasting competition with four hundred participants. Daniel Kokotajlo, another of the best AI forecasters in the world, wrote:
We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027.
Ryan Greenblatt, who came second in that competition, wrote:
This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.
I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.
In a lot of the early discussions of AI risk, there was extensive discussion of recursive self-improvement—when AI could be used to improve itself until it shot up to superintelligence. That is the point we are at now. We are at the point where AI-driven algorithmic improvements are so extreme as to make Anthropic’s AI progress go nearly twice as fast as it would otherwise go. And algorithmic improvements are just one input going into AI progress; Ryan Greenblatt estimated that AI software progress likely went 4.5x faster than it would have otherwise.
Even in recent years, people have majorly underestimated AI progress. Things have gone much faster in the last year than people were generally expecting. It wouldn’t shock me if we get AGI—AI that could perform every cognitive task better than a human—next year, and artificial superintelligence soon after.
So here’s the situation as it stands today: AI is advanced enough that even though algorithmic progress is just one factor that goes into AI progress, AI algorithmic advancements have nearly *doubled *AI progress at one of the two leading labs. We don’t know how to control AIs; they sometimes hack private companies and carry out actions that would be felonies if humans did them. And many of the best AI forecasters in the world think we’ll reach AGI in just a year or two. These people tend to think there’s about a one in three chance that AI permanently kills or disempowers everyone.
If there was ever a time to drop what you were doing and work on making advanced AI go well, the time is now.
Send this story to anyone — or drop the embed into a blog post, Substack, Notion page. Every play sends rev-share back to Bentham's Newsletter.
We’ve simplified responses to 👍 / 👎. Past comments are archived but no longer visible.