This is a guest post by ACX grantee Josh Morrison of 1DaySooner.
AI superforecasters are coming. Within a few years, the average person will have the equivalent of a Nate Silver in their pocket, ready to predict whatever question they can think to ask it.
Scott wrote:
Savvy institutions will cede some of their strategic thinking to AI. Before starting a new project line, smart businesses will ask the superforecaster AIs how much money it will make (their investors will definitely be asking!) Smart political consultants will ask the superforecaster AIs about their candidates’ chance of winning conditional on running this or that ad.
The government isn’t usually classified as a savvy institution, but we might hope that parts of it will seek AI forecaster advice. Probably defense analysts will include in their PowerPoint presentations some fact about how AI forecasters say their new fighter jet design is more likely to find a use case than some competing fighter jet design.
I don’t know anything about fighter jets, but I run an advocacy group that works on reforming clinical trials and the FDA, and I think cheap superforecasting could revolutionize our field.
Ideal regulation would be a “government of laws and not of men”, where regulators communicate clear standards to companies, and companies determine how best to meet those standards in their unique situations. The FDA can’t currently do this. Medicine is so complicated, with so much money at stake, that any “objective” standards subtle enough to separate good from bad medications could be rules-lawyered by loophole-trawling pharma companies. Instead, the FDA relies on human judgment: a team of staffers reviews an application, and a bureaucrat makes the final decision. Companies don’t know what the experts and bureaucrats will look for, and a single rejection could devastate their bottom line, so they err on the side of caution.
On paper, the FDA’s regulations are reasonable and aim to promote innovation. In reality, the above considerations incentivize companies toward conservatism, sticking to exactly the same processes that have worked before and that the FDA’s experts are most likely to understand. For example, in theory a company only needs to file one Investigational New Drug application with the FDA before beginning their study, and if the FDA doesn’t veto their idea within thirty days, by default the company can proceed (and then submit similar notices when they want to amend the study). In practice, at each phase, companies acquiesce to the FDA reviewers’ “non-binding” questions, comments, and criticism, leading to long delays before trials even start. They hope that if every choice is made by an FDA official – even if it makes the study worse or introduces substantial delays then later, during the drug approval phase, FDA reviewers won’t penalize them for making other choices. This process adds months to years of delays, and millions to billions of dollars in costs.
This subjectivity hurts patients as well as drug developers; the same reviewers who veto drugs for subjective reasons can approve them for subjective reasons too. In 2021, the FDA faced widespread criticism for approving Biogen’s expensive Alzheimers’ drug Aduhelm after its own expert panel near-unanimously voted against. Investigators documented an “unusually close collaboration” between the FDA and Biogen during the approval process. But given the incentives, it’s hard for companies not to get enmeshed with their regulators over the course of their trials.
Everyone here is acting rationally: the FDA wants to avoid shoddy trials, and the companies want to avoid getting their flagship drug because a reviewer didn’t agree with how they ran their study. To break out of this equilibrium, we need a judgment process that combines the intelligence of an expert with the objectivity of a simple ruleset.
Forecasting could fill this role, turning subjective judgment calls into objectively observable bets about the future. If the FDA makes quantitative predictions about its decisions, then drug developers can do the math on their own and aim at the objective standard (“what will the forecast say if I do this?”) instead of a subjective one (“what do FDA reviewers want?”).
A Path Towards Forecasting At The FDA
Pharmaceutical forecasting is untested, and the FDA is a conservative organization. Decision makers would not and should not uproot their usual process in favor of a forecasting-based one. We envision applying forecasting to the FDA gradually, increasing its prominence only after low-commitment applications have visibly succeeded.
In the first stage, the FDA could add forecasts to its existing decisions. For example, when a pharmaceutical company submits an Investigational New Drug application, the FDA’s review would include forecasts about the likelihood that someone dies in the study, or what the cause of death would be. When the company runs a human challenge trial, the FDA should forecast the risk of infection to the participants and likely efficacy of the treatment. When they apply for final approval, the FDA should forecast how many people will take the drug (if approved), what side effects will be observed in post-marketing surveillance, and what the effect on outcomes (for example, how many people die from the relevant disease) will be. These are all factors which the FDA should be considering already; the first step of forecasting would be to make them explicit, so that they could be checked against future data.
At first the forecasts will likely be imperfect and not ready to be relied on. But as trial results and real world evidence accumulate, the forecasting system will improve and be made reliable. The FDA currently has a program, Sentinel, that uses real world evidence to identify safety signals among approved products1. For a forecasting system to be most effective, it would need to check its predicted product effects against what is actually observed in a more advanced version of Sentinel or another similar program.
This summer, as part of its Operation Trialblazer clinical trial reform initiative, the Trump administration announced a pilot program to speed up phase 1 trials. Incorporating forecasting into that pilot would be a concrete way of implementing this initial stage.
The FDA could either employ human forecasters, partner with an AI superforecasting company, or create their own in-house software; results from this first stage would be used to calibrate its effectiveness.
As results from these preliminary forecasts built up, the FDA’s forecasting tools could be made public, and drug companies could look at results to better understand the prospects of their own trials. They might see that in the past the FDA approved studies with a 1/1000 chance of causing a death, but not a 1/100 chance, and calibrate their own appetite for risk appropriately. The FDA itself would likely go through a similar process, as experts tried to maintain an appearance of consistency with their past decisions.
In a final stage, after the forecasting system had been validated, quantitative forecasts could become legal triggers. For example, if a historically reliable forecasting system predicted a >95% chance that a drug’s benefits would be >5x higher than its negative side effects, that product could be automatically approved. Or clinical studies could automatically go forward where they were predicted to have less than a certain level of risk. Pharmaceutical companies could have access to the system, ask it their questions (“What would you say if we proposed Study Design X? What about Study Design Y?”) and have a simulated FDA that responded to their queries instantly, cooperated with them to design the best study possible, and always made (and stuck to) predictable decisions.
While such a system seems a long way off, its value would be immense. It would be self-improving, so risks to research participants would decline over time as ability to predict hazards rose. Because risk would be lower, trials could proceed with fewer burdens, reducing the cost of medical research. And because one could better predict the effect of a successful experiment on the forecast, the studies we chose to do would be more efficient and better targeted, allowing drugs to reach patients sooner.
Sentinel is a distributed data network where the FDA sends queries to its data partners, who run the analyses locally and send the results back to the FDA. This means that no medical data has to be held by the FDA, reducing the risk of releasing identifiable health information. To best make real world verification of FDA forecasts work, Sentinel would have to be expanded to analyze de-identified but full electronic health records for a significant portion of Americans.
