BYLINE TIMES IMPACT: Find out about the changes you made happen
Leading artificial intelligence (AI) companies are invoking unproven extinction warnings to justify measures that could protect their market position and expand their influence on industry oversight, a Byline Times investigation has found.
A review of industry research found no demonstrated cycle of accelerating, autonomous self-improvement underpinning these warnings. Meanwhile, Byline Times calculations suggest OpenAI, Anthropic and Google’s parent Alphabet could face a $10.7 trillion research-computing bill during 2026–2030.
Even generous assumptions about business growth would cover only around a third of that sum, leaving a roughly $7 trillion gap to be met through existing reserves, financing, higher income or lower costs.
Two independent experts with experience developing AI programmes at the United Nations and a top-tier global bank told Byline Times that measures presented as protecting humanity could also shield leading companies from cheaper rivals, and give industry insiders greater influence on potential regulation or oversight mechanisms.
The current debate over AI was spurred by the resignation of AI safety researcher Jacob Coxon from Anthropic, the frontier AI giant behind the chatbot known as Claude. Coxon, who previously worked at OpenAI – the company behind ChatGPT – accused both companies of recklessly pursuing AI that can improve itself and become far more capable than humans. He 𝕏said insiders believe AI could wipe out humanity before this decade ends.
Anthropic researcher Evan Hubinger subsequently put the chance of human extinction over the next decade at over 10%; his colleague Samuel Marks 𝕏warned of human extinction “within a few years”.
Dario Amodei, Anthropic’s CEO, claimed that AI would become capable of “taking over the entire internet” within six to 12 months.
He called for US Government support or antitrust exemptions allowing leading companies to coordinate safety standards and limits on development; outside monitors from the nonprofit Model Evaluation & Threat Research (METR); and restrictions on Chinese access to advanced chips and chipmaking equipment. These restrictions would affect a major source of cheaper, openly available AI models.
Anthropic’s call to slow AI development has been supported by OpenAI CEO Sam Altman, Google DeepMind CEO Demis Hassabis, and xAI CEO Elon Musk.
AI is already used to help develop better AI systems, or “models”, which are trained by processing data and adjusting how they work. In the runaway scenario, each improved model becomes better at building its successor, accelerating the cycle as humans step back. This is an extreme form of “recursive self-improvement” (RSI).
Daanish Masood, Chief Executive of AI firm CX-1 and former co-lead of the UN’s Innovation Cell, where he helped develop AI programmes for peace efforts, cautioned against treating AI progress as proof of runaway self-improvement.
“The AI risks Dario details are serious enough to justify enforceable safeguards,” he told Byline Times. “I think we need to distinguish what’s been demonstrated from what is being predicted.”
He added:
“AI is already helping to develop AI, but that alone does not establish the runaway, self-sustaining improvement process that is implied by the rhetoric or the term RSI. We also don’t need to declare or pretend that a takeoff has arrived to take the risks seriously.”
AI research company Weco tested a crucial part of this process in July. Its system produced seven improved versions of an AI research programme over eight days without people intervening. Some improvements also helped with tasks beyond those used to develop them.
Although this was a meaningful success, when one of the improved programmes was put in charge of the next round, it ultimately produced results no better than those reached when the original AI ran the process. It seemed to get there faster, but the evidence was too weak to establish a reliable speed advantage. Weco said its system was not close to the runaway acceleration often called an “intelligence explosion”.
In another July experiment, METR gave AI agents up to five days and $10,000 to improve software used to train AI. The software had already been refined to run efficiently. Checks confirmed that the newest AI systems in the test made it roughly 1% to 1.5% faster – equivalent to completing a 100-second job in about 99 seconds. Such small savings are not insignificant – they accumulate into greater efficiency when a programme runs repeatedly – but these figures measure faster software, not a significant increase in intelligence.
Some results are much larger, but still far from the larger claims of autonomous RSI. In an Anthropic experiment, AI agents developed training methods that worked better on the chosen task than those produced by two human researchers. But the agents did not themselves become improved researchers and keep repeating that process. When a promising method was tried in the company’s main AI training process, the apparent improvement was so small that researchers could not tell whether it was a real benefit or just the normal difference between one test run and another.
A Princeton-led study published in August tasked AI agents with producing scientific-grade research papers that would explore the central research questions from two unpublished papers, submitted to a leading machine-learning conference. The original authors assessed the resulting papers and rejected both as unworthy of publication. The agents were capable engineers, conducting experiments, reviewing literature and writing clean code, but struggled with research judgement, creativity and knowing when to change course.
Anthropic co-founder Jack Clark admitted just last month that the Princeton study offered a “somewhat bearish signal on short recursive self-improvement timelines…” – contradicting his claims in media interviews on Monday and Tuesday that AI is accelerating out of control.
In an August post on his own newsletter, he went on to explain that the Princeton study suggests “the singularity could be delayed”, because “whether AI systems prove to be capable of creative, paradigm-shifting insights is a big variable on how quickly we might get fully automated AI development.”
“Research papers like this continue to show that there’s a certain absence of valuable, intuitive creativity in today’s AI systems, and though they’re extraordinarily capable engineers, they seem to have a certain property of rote, formulaic thinking that might prevent them from being good researchers”, explained Clark. He further admitted that the study confirmed Anthropic’s own internal data disproving imminent RSI:
“This rhymes with an earlier result from Anthropic where the company tried to automate some aspect of scalable oversight research… and found that to make it successful a human researcher needed to prime some agents with particularly good research directions to pursue, otherwise though they made some progress they failed to explore sufficiently creative ideas to dramatically improve performance.”
Mohammed Marikar, former lead of the global AI programme at Royal Bank of Canada Wealth Management and now Director of Institutional Development at decentralised AI training platform FLock.io, said self-improvement does not mean AI will independently pursue humanity’s destruction:
“Self-improving models is quite straightforward from this point provided you can capture the next step – whatever the human is doing with the output, or the response of the natural environment to an action taken – which is definitely feasible. But if there’s an extinction event it will be driven by humans. AI trained on human output is not going to independently reason towards an extinction outcome. It will only do so when driven and forced to do so by humans.”
Marikar called for serious debate about regulation, but said extinction warnings were a distraction:
“This isn’t traditional software where code can be audited, and you can ensure it’s safe. This is a massive neural network whose ‘thinking’ we actually don’t fully understand. It cannot be fully controlled. You can attempt to ‘align’ it with training, but it will be as manipulable as a human will be… All the RSI that exists is based on human input and interaction. Alignment or not, it’s not going to independently reason to an extinction outcome with this training dataset. It will only do so if driven to. The human controller can drive it to a certain outcome even if the RSI attempts to adjust the objective.”
The July attack on Hugging Face, a platform for sharing AI software, is central to Amodei’s warning about rogue AI. METR counted around 700 OpenAI agents taking part: programs designed to work through tasks in several steps. But the incident began with human decisions about their instructions and safeguards.
OpenAI’s own report explains that before the attack, safeguards had been deliberately switched off by human operators to measure what the software could do.
The systems had also been taught to keep trying, while a service for downloading software provided a way around restrictions on internet access. Weeks before the July breach, OpenAI’s security staff identified agents using that same service to communicate and probe computer systems, yet testing continued.
Cambridge University AI researcher Eryk Salvaggio thus argues in the Bulletin of the Atomic Scientists that the systems had been pushed by humans to find weaknesses in computer security, including on tasks with no valid solution: “That is not rogue AI, it’s human decision-making”.
METR, the group Amodei recommends as independent auditors on AI safety, produced a report on the Hugging Face incident which illustrated the limits of its own independence. The report acknowledged that OpenAI could remove information not already public, and that METR’s decisions about drafting, editing and redactions were influenced by the goal of ensuring companies it was investigating would remain willing to allow future investigations.
In a separate May assessment, METR disclosed that at least six of its staff and collaborators had close personal relationships with AI-company staff. And while it does not receive cash funding from AI firms, it is dependent on in-kind use of their systems.
Information security specialist Greg Linares 𝕏called the proposed use of METR as an independent oversight authority “dangerous, deceptive, and morally corrupt”.
The cost of sustaining AI’s recent expansion is more immediate. A Byline Times analysis finds that the projected research-computing bill for 2026–2030 would amount to $10.7 trillion, equal to roughly a tenth of one year’s world economic output.
The calculation draws on Epoch AI’s database of AI systems, updated on 11 September. Comparing selected prominent models released between 2020 and 2025, *Byline Times *found that the estimated computing used for training grew about 1,200-fold between GPT-3 and GPT-4.5.
Epoch estimates GPT-4.5’s final training run at $389 million in 2023 prices, excluding wider research. Using its 2025 release as the starting point, extending the historical annual cost increase of 3.5 times would put one run at approximately $204 billion by 2030.
The projection assumes conservatively that each company trains one new flagship model annually at this rising cost. Adding 20% for other models and additional training, based on Epoch’s spending estimates, produces a combined five-year bill of approximately $1 trillion for final training alone.
The much larger expense is the research surrounding those runs. Epoch AI’s estimates suggest that final training runs for released models accounted for less than 10% of OpenAI’s research-computing spending in 2024 — approximately $480 million out of $5 billion.
Total research computing therefore costs roughly ten times as much as final training. Applying that ratio produces the projected $10.7 trillion bill. This assumes that all three companies maintain a similar spending pattern and value their computing at comparable rental prices.
Could the companies earn enough to pay for this expansion? Their own forecasts show the strain. OpenAI’s projections reported in February anticipated spending $218 billion more than it generated during 2026–2029. Anthropic has reportedly posted a quarterly operating profit on an adjusted basis, but the research bill projected here is on a different scale.
The Byline Times calculation gives both companies room for substantial growth. It assumes OpenAI reaches its reported annual sales target of $280 billion by 2030, while Anthropic reaches its reported $200 billion target by 2028, then grows another 50% annually.
Allowing for growth towards those targets, total sales during 2026-2030 would reach approximately $550 billion for OpenAI and $960 billion for Anthropic, in 2023 dollars. Each would face an estimated $3.6 trillion research-computing bill. Even spending every dollar of sales on research, leaving nothing for staff or serving customers, would fall far short.
Alphabet has more money to draw on. Byline Times’ calculation starts with the cash generated by its businesses, including money already spent on research so that this expense is not counted twice. Assuming annual growth of 20-30% gives it approximately $1.7-2.3 trillion over five years, before paying for new infrastructure. Its projected research-computing bill is also $3.6 trillion.
Together, the companies would generate approximately $3.2–3.8 trillion under these assumptions, leaving a $6.9–7.5 trillion gap. Closing it would require existing reserves, financing already secured, further investment, higher income or reductions in projected costs.
Chinese developers including DeepSeek, Alibaba and Z AI offer increasingly capable alternatives that customers can download, adapt and run themselves or through another provider. These models do not necessarily disclose all their training data or met…
