I disagree with Henry Farrell’s subheadline here, which I think contains some other-side-of-the-Atlantic understatement. It is not that there are other ways of thinking about “AI” than “Digital God”, it is that it is difficult to think of a way less helpful for coherent thought to think about “AI” than to think of it as a “Digital God”.
But I think he is right in the rest: That means that the important and live questions are about what metaphors—what abstraction layers, throwing away information so that the problems become things we can grasp and think about—are useful right now? Henry Farrell’s “MAMLMs are a socio-cultural technology” is one very useful one. The “‘AI’ is a normal disruptive Schumpeterian GPT” is a second. What are the others that we need to hold in our neocortices right now?":
<https://www.programmablemutter.com/p/machine-god-metaphors-eat-your-brain> <https://www.programmablemutter.com>
Here are three propositions.
Metaphors are the fundamental abstraction layers in modern AI debates.
We are in a moment where you could plausibly argue that Eliezer Yudkowsky and Nick Land are the most influential policy intellectuals in the US conversation. Just look at the state of the two of them. This is not a good thing.
The collision of (1), (2), and recent events means that “machine gods” have now entered the discussion. This is a problem. Abstractions necessarily throw information away. The information that the “machine gods” metaphor/abstraction dumps is crucial to navigating the moment we are in. If public and policy debate stays centered around “machine gods: good or evil?” we are in for a bad time.
I should acknowledge that I (and co-authors) absolutely have our own dog in this fight. Indeed, I think that the “AI as social and cultural technology” approach that we have developed is a much better alternative. The final part of this newsletter will talk about a new piece by a mathematician that builds on this approach. Equally, my claim is not, actually, that our alternative approach should become the sole dominant way of thinking about AI. What I actually want, along the lines of my previous post, is a much greater degree of pluralism in the defining debates around a technology that no-one, including its creators, seems to understand particularly well or to be capable of usefully predicting.
As far as I can tell, the above Twitter exchange is the source of the most recent outbreak of Machine God Thought. People who know Daniel Kokotajlo tell me that I’d like him, and that we could have good conversation; I imagine they are right. I’ve found most of the rationalists I’ve met to be decent human beings; in many cases, unusually so. Equally, I think that there is something terribly limiting to the ways they talk about AI, and contend that it’s a real problem that debate is still accelerating past the event horizon towards the very heart of Singularity thought.
The two papers Kokotajlo mentions here, AI 2027 and AI 2040, lay out futures that are roughly identical in their predictions about what AI becomes (godlike; superhuman). Where they differ is in their assumptions about how well aligned AIs are with human goals. In the scenario where AI develops more or less unconstrained (thanks in part to geopolitics), AI breaks out of containment, and humanity is likely either Corgi-ized or goes extinct. In the scenario where AI development is delayed, tamed and made subject to “mutual compute destruction,” 1 men and women come to have gods as servants, and start spreading out to nearby galaxies in the next two decades.
Neither future seems to be at all convincing to me, but they are most certainly convincing some people. Casey Newton and Kevin Roose are partnering with NPR to create the “Machine Gods” podcast as part of “a bold new era of content strategy for the network.” In his newsletter, Newton thanks the fates that he and his collaborator were a little too late to tech podcasting to be “beguiled” by Sam Bankman-Fried, and suggests just two paragraphs later that critics of Roose’s famous shoggoth freakout were “previewing the ways in which the commentariat would continually twist itself into pretzels to deny that anything important was happening with LLMs.” Indeed.
Things are no better elsewhere. In Washington DC, the AI policy debate is riven by a bitter dispute between those who take the Kokotajlo perspective (slow it down before we all die!) and those who want to let innovation rip so that we hit the technocapital singularity as soon as possible. The one side owes its framing to Eliezer Yudkowsky’s MIRI institute. The other builds on Nick Land Thought as filtered through the more immediate needs of Marc Andreessen. While I’m definetely more inclined toward the former than the latter, on the grounds that we do indeed need to have serious conversation about regulation, I think it would be terrible to have that conversation starting and stopping with the priorities of the labs.
Finally there’s Scott Aaronson, who I used to rely on for valuable debunkings of policy and media folk when they got over-excited about quantum computing. It is fair to say that Scott is not in a debunking mood these days, now that OpenAI has solved the Navier-Stokes problem. Witness the below:
The Age of Wonders and TerrorsTwenty years ago, when the idea of AI taking over the world in our lifetimes still struck most of us as the unconstrained fantasy of those who knew too much science fiction and too little science … update on the fact that
the wild prophecies have come true. … It seems to me that the Singularityhas already started; it’s just wildly unevenly distributed.… For anyone who says AI doom sounds like an apocalyptic religion, that the rationalists/Singulatarians seem like a Bay Area cult, that Eliezer Yudkowsky gives off the vibes of a messianic prophet: yes, yes, and yes. But crucially, today you’re no longer being asked to believe in arguments and extrapolations,
but only in the front-page news.Accepting the reality of the coming machine godafterit’s solved Navier-Stokes and dozens of other longstanding open math problems (while dramatically ramping up in capability every month), is sort of like accepting Jesusafterhe’s returned to earth on the gleaming cloud. It’s the epistemic bare minimum.Yes, there’s still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell, there’s no longer any real uncertainty that it’ll all mostly revolve around AI, and the extent to which we succeed or fail at directing its power toward human flourishing.
So why do I think that the machine god approach, far from being an “epistemic bare minimum,” is a pernicious brain-devourer? The answer is that the metaphor shrinks rather than expands your horizons of the possible, forcing you to focus on the single overwhelming question of what Machine Jesus is going to do as soon as he descends from that gleaming cloud. Is he going to usher in paradise on earth? Or is he actually a shoggoth in a Jesus-mask, condemning us to enslavement or destruction? Once you have accepted that Machine Jesus is in the process of descending, all questions besides determining his intentions for humanity seem irrelevant.
I’ve written before (with colleagues and on my own) about why I think this whole way of thinking is utterly wrong-headed. But maybe I’m wrong. Take three somewhat inter-related possibilities: p1, that humanity is not the sole beautiful flower of the cosmos and that greater than human intelligence is perfectly possible, *p2, *that high intelligence is effectively impossible without strong intentionality, and *p3, *that current AI architectures are indeed the royal road to achieving proper generalizable superintelligence. I personally rate *p1 *as extremely high, *p2 *and *p3 as very low, and p1p2p3, *which is the most obvious state in which you might be worried about the AI alignment problem, as very low indeed. But who is to say that I might not be wrong? Equally, I, or someone who is skeptical of Machine Jesus for wildly different and incompatible reasons than my own, might be right. And if that is possible, then we ought not just be obsessing about whether we “succeed or fail in directing” His Awesome Power towards human ends. We might want to worry about other questions too, or instead.
There is a discussion of the solution of Navier-Stokes, the miracle that brought Scott to embrace the true faith, in a guest-post on Terence Tao’s blog, which usefully illustrates the point. It also has the benefit of being (sort-of) pre-registered. This discussion is specifically attractive to me, because it builds on arguments about AI as a social and cultural technology. Again, I don’t think that you need to believe that it is right on the merits to accept the more limited point that I want to make, which is that perspectives other than the machine god one reveal other questions and problems that we ought very reasonably want to be concerned with.
In his guest-post, the mathematician Nestor Guillen writes that he anticipated the solution of Navier-Stokes, but that his reaction was not that human mathematics was in danger of being replaced. Instead, he argued that it validated the entire enterprise.
the bulk of the essay was done in August and shared with a few close friends for feedback. Earlier this week, Tristan Buckmaster
[announced]a breakthrough finding, done in collaboration with Levent Alpöge, of a finite-time blow up for 3D incompressible Euler with forcing. He also made serious allegations of misconduct by OpenAI (see announcement). The next day, OpenAI announced a finite-time blow up result for 3D Navier-Stokes with forcing, obtained from their internal LLM. … I have decided not to change the opening of the essay and instead make a note of the news here.… What would be the significance if one day we woke up to the news that output from a Large Language Model contains a definitive answer to the question of blow up for the incompressible Navier-Stokes equations, and to the question of where the zeros of the Riemann zeta function lie?
Many of my fellow mathematicians consider this a kind of nightmare situation, and I think this is a very valid position. Here I would like to make a different case, however. Personally, I have come to believe that the news of a Large Model output containing an interesting, insightful, novel mathematical idea can and should be received
as a source of communal pride for mathematics and for mathematicians.
Why is this so? Guillen says that:
Humans have always lived in a complex world and their survival has hinged on their ability to process information in amounts beyond what any individual can handle — and it is likely this has been so nearly as long as humans have existed. Herbert Simon’s concept of “bounded rationality” captures this well …
Simon saw that one implication of bounded rationality is that humans have to (and do) develop systems to process information, as a group. We see that humans have built states, stories, bureaucracies, markets, libraries, and more. These are cultural and social technologies. They are means by which humans (not individually but as a group) navigate amounts of information so vast that they cannot be handled by any individual on their own. …
LLMs … are “wrappers” around an older cultural technology: the scientific literature. When you ask an LLM a question about the isoperimetric inequality, the ABP maximum principle, or a PDE, you are effectively interacting with every person who has thought about and worked on those things — provided the data the LLM was trained on contained those contributions in one way or another.
Unsurprisingly, I think that this is largely correct (I *do *think that you can make the cultural and social technology argument, while also being freaked out about the consequences of AI: more on that in my next post). But even if it isn’t, it highlights different, and I think, important questions than the machine god nexus.
Specifically: if you think of AI as a cultural and social technology, your first reaction will not be to prostrate yourself in anticipation of the coming Age of the Gods. Instead, you are likely to treat fields like mathematics as some combination of individual and collective human intelligence, constructed on top of technologies that guide and direct that intelligence in particular directions. So when a new technology such as AI comes along, that reshapes the ways in which mathematicians draw on their shared intelligence, you will have questions to ask. What are the current processes through which existing knowledge is disseminated and new knowledge is created? What are the incentive structures for dissemination and creation, and how do they interact with each other? How have processes and incentives been shaped by previous and existing cultural and social technologies? How might these processes and incentives be changed by what is happening now?
And so on. You can see one way in which you might extend this in Ben Recht’s recent post, which suggests, if I understand him right, that mathematicians really ought to rethink how much of their field was already based around benchmarkmaxxing (prizes and challenges) even before AI came on the scene. Tao’s argument, as quoted in the * NYT*, points in a similar direction.
[Tao] compared OpenAI to a wilderness guide who finds a path to a hidden waterfall. “This has some value,” he said. “But once someone shows a specific path to the waterfall, people just take that path. They don’t spend as much time looking for other paths.”
I am not even an amateur mathematician, so am very much open to correction, but I understand Tao to be saying something like the following. Automated processes of discovery have obvious advantages in pointing the direction to new things. Equally, they push the needle from ‘explore’ to ‘exploit.’ Apparently aimless noodling around by humans can have benefits when it opens up unexpected views from Serendippo . More generally:
solving problems is only a tool and proxy for a…
