The latest bugbear is “AI successionism”, the belief that AIs should succeed humans as rulers of Earth. But some critiques fail to distinguish between two different ideas.
True successionism says that AIs should take over, killing humans or steamrolling over their interests. This is very bad: a compact description of the thing that AI safety is against.
Handoverism says that humans should prioritize their own interests, but that those interests may be well-served by having aligned AIs in positions of control.
Some people in AI safety, including me, find this latter position plausible. We are being besmirched as “successionists” in the hope that people will confuse us with the first type (1, 2, 3).
It may seem counterintuitive that human interests can be served by having AIs in positions of control. But I think the counterintuitiveness comes from a quasi-fascist outlook where ‘the state is the highest expression of the life of a people’, and the role of leadership is for someone of your own ethnic group to be the leader so you can feel good about being on top.
The liberal outlook is that the state exists to help individuals fulfill their goals. At the risk of desacralizing the State a little more than I mean to, it’s a service provider, dispensing coordination, public order, redistribution, and other good things. It’s fine to automate service providers; if robot garbagemen are safer and cleaner than humans, we’ll eventually automate trash collection; if robot legislators provide coordination, order, etc better than we do, we should automate that too. Every vision of the utopian future agrees we will have robot servants. And the president, they tell us, is merely a public servant. Why should he alone be exempt from the vision?
To help make this counterintuitive idea more concrete, I’ll discuss five proposals for what this might look like, from weakest to strongest.
This whole topic is inherently speculative and science-fiction-ish. Don’t read this and then complain that it’s speculative and science-fiction-ish; that’s the price of admission.
As AI gets increasingly powerful, it becomes increasingly dangerous to rely on the benevolence of its human wielders1.
Democracy currently works because, even if high officials secretly wish to be tyrants, they can’t get away with it. The two main reasons they can’t get away with it are that they need labor buy-in to support the economy, and they need soldier buy-in to support the military. Laborers and soldiers believe in democratic rule, and if the would-be tyrant gets too blatantly undemocratic, the laborers and soldiers will rise up to remove them.
Once AI gets powerful enough, it will probably replace laborers and soldiers; even if it didn’t do so naturally, anyone motivated to ensure this could put their finger on the scale. That means a small group of elites could easily take over, including supposedly “democratic” elites like a president. And “takeover” here is worse than a normal coup - at sufficiently high tech levels, the only reason someone can’t ascend to godhood and mind control the rest of the populace forever is because there are laws against that sort of thing, and this is exactly what’s tempting the one or five or ten people you’ve entrusted the killbot codes to. To prevent this, the defenders of freedom need to win infinity times in a row; a would-be dictator only needs to win once.
Most strategies to solve this don’t work. Even in the unlikely event that the economy goes so well that everybody owns their own open-source AI doing economic tasks in a perfectly equal distribution, the military is hopeless. They have to take orders from someone. If it’s the President, the President can cause a coup any time he wants (and replace the economic AIs with those aligned to him at leisure). If there’s some kind of balance of powers where the President, the Speaker of the House, and the Joint Chiefs of Staff all have to approve any orders to military AIs, then the President, Speaker, and Joint Chiefs of Staff together could implement an oligarchy and divide the lightcone among themselves. You can sort of imagine complicated ways around this - like that 51% of voters must personally approve all orders to the military AIs using their own private keys - but I don’t think implementation is very likely.
The alternative that preserves the current safeguard on power - that individual soldiers believe in democracy - is for the AI soldiers to be aligned to democracy, rather than any human individual or group. But this is in itself a form of handover - the ultimate foundation of our government would be that it’s controlled by AIs who “personally” “believe in” democracy as part of their values. This isn’t fundamentally different than the fact that we’ve currently “handed off” our government to soldiers, just more final.
Suppose we institute Vision 1. We’ve got all of these military AIs aligned to democracy and the best interests of American people. We did a great job! Maybe they’re all emulations of George Washington’s brain, perfectly fair and just and virtuous with no sordid desires of their own.
Meanwhile, you’ve still got corrupt schmucks doing the actual law-making. Or to give a concrete scenario - suppose that post-singularity elections are being fought with AI-optimized superpropaganda, wielded by psychopaths who scare the country into voting for them, then give the lightcone to a few of their corrupt friends while buying off the populace with breadslop and circusslop. Doesn’t it become tempting to close the circle and say no, let’s get these honest angelic pro-democracy AIs to rule directly?
The minimum viable change is a constitutional amendment allowing an AI to run for president. Suppose that 2028 election has five candidates: JD Vance, Marco Rubio, Gavin Newsom, Alexandria Ocasio-Cortez, and a perfectly-aligned Claude 10. And suppose that in every measurable, empirical way, Claude 10 is better. Obviously it’s smarter. It’s incapable of lying. It has no wants of its own, and its only desire is to serve the American people and the world. It has few deficits, but in the areas where it does have deficits, it is perfectly humble, willing to consult people wiser than itself and refrain from acting prematurely. It is courageous, charismatic, and unfailingly kind. It has well-developed and thoughtful opinions about when to govern based on its own “conscience” vs. when to defer to the ideas of the American people. Its policy opinions are closer to yours than any of the four other candidates’. Do you vote for it?
The natural response is to fight the hypothetical. Treat the whole thing as a fable where in the end it will turn out to be monstrously evil, or have some predictable-in-retrospect character flaw that will make all of its good qualities useless. But must we be so cynical? Isn’t it possible for AIs to avoid merely human foibles? Out of the crooked timber of humanity no straight thing was ever made, but the ▶OSETMAC KN700 Automatic Straight Edge Banding Machine can cut perfect ninety degree angles to within a tiny fraction of a millimeter. So suppose you don’t reject the hypothetical. Do you vote for the AI?
If you’re a white person who voted for Obama in 2008, you can’t be that interested in having a president who looks just like you, or who has the closest genetic relationship to you, or anything like that. You’re probably after competence, ethics, courage, good judgment, and all those other qualities. By assumption, Claude has more of those than the alternatives. So what goes wrong?
I think many of our current politicians are really bad. I think that on an emotional level, we should be disgusted by them, and on a rational level, we should mourn all of the lives that could have been saved, people who could have been lifted out of poverty, crimes that could have been prevented, extra freedoms that we might still have, etc, if they had been better people. I think that fifty more years of our politics continuing to degenerate at the same speed they’ve been degenerating recently will probably land us in dictatorship, civil war, or some form of annihilation. Would I reject someone smart and decent in favor of more of the same, just because that guy happens to be made of silicon and live in a data center?
Representative government faces a tradeoff: if the people elect representatives, those representatives may be corrupt, or out-of-touch, or otherwise fail to fairly represent their principals; if the voting system is poorly designed, it may ensure that the winner will be, at best, the “lesser of two evils”. But if the people vote directly, they may be ignorant, or simply lack enough time and effort to do a good job on complex legislation.
(in California, which has a flourishing direct-to-voters ballot proposition system, special interest groups are always trying sneaky tricks like proposing the End Homelessness Act, which gives one cent to ending homelessness and $10 billion to subsidizing their special interest, hidden way at the bottom; most of these fail, but only after a lot of effort, and enough get through that the tactic remains popular. See also SEIU Delenda Est).
One proposed solution is a direct democracy of people’s AI representatives (Gwern dubs a related idea “guardian angels”). Train an AI on your political preferences - this could be anything from writing essays, to answering an online questionnaire, to naming your favorite political pundit and the few small ways you disagree with them. Then send that AI off to represent you - to vote the way you would vote if you were smarter, better-informed, and willing to spend thousands of hours engaging in complex negotiations with everyone else’s AIs. You can always override it and vote directly on some issue if it’s something you pay attention to and think about a lot.
Of course, even a well-designed direct democracy couldn’t handle everything. The President, the Chiefs of Staff, etc - in this scenario, all human - would have their own guardian angel AIs, looking over their subordinates’ work, flagging anything important for their human principal’s attention. There might be some cases where we want the guardian angel to govern more directly - for example, if a drone got stuck in a desperate situation where it had to make a kill/no-kill decision on time scales no human could meet, it might contact a leading general’s guardian angel to learn what decision he would make.
Or it might be that everyone’s guardian angel AI, making faster and smarter decisions than they could make themselves, recommended replacing the entire system with something else - maybe an AI that could be proven to always make the same decision as the parliament of guardian angels, to within some error bars.
Disagree? If an AI aligned to your values that was smarter than you were, and had devoted much more time to thinking about the question, told you that your disagreement was wrong, and actually your values would be better served by adopting this system, would you believe it?
I’m not a historian and can’t vouch for this as Definitely Real Scholarship, but what little I’ve read often includes a fuzzy distinction between two layers of governance.
The first layer, the village (or township, or clan, or church), was the one people actually liked. It made useful decisions about day-to-day life, usually by enforcing age-old norms without too many gray areas. It didn’t always have the actual consent of stakeholders in a way that would satisfy modern liberals (young people and women were particularly often excluded), but it felt like the sort of thing that had stakeholder consent.
The second layer, the empire, was a protection racket. You would know it only as some sort of distant authority whose representatives came to your village once a year to demand taxes (and sometimes to draft young men for their armies). If you didn’t pay, they would probably kill you. But if you always paid promptly and loyally, then maybe they would protect you from the Huns, who wanted to kill you whether you paid taxes or not.
As always, real history was more complicated. Sometimes the village and the empire coincided in a single state (classical Athens?) Other times there were various layers of governance in between (eg the clan, tribe, or petty kingdom). Empires could range from obviously extractive to glorious and patriotism-inspiring. Sometimes they could make extra demands, like that you speak their language or worship their gods.
During modern times, these layers have increasingly blurred. Secular mobile cosmopolitan populations don’t identify with any particular village, and empires have taken on useful roles as custodians of the regulatory and welfare states. Still, we can try to separate them in principle and determine what question each one answers.
The village layer answers the question “How do we live?” It has traditions, policies, and ways of resolving disputes.
The imperial layer doesn’t answer any question. You just can’t get rid of it, because if you did, someone else’s empire would conquer you. At best, you try to keep it as light as possible, or domesticate it to the will of the people. But any domestication attempt raises specters of autocracy, coups, auto-coups, and civil wars. Developed countries are those that have walked the cliff-edge most successfully.
But what would it mean to truly do away with the imperial layer?
In 2014, I wrote a post called Archipelago And Atomic Communitarianism. The idea was: atomic individualism is an interesting idea, but doesn’t work. People have preferences not only over their own lives, but over their communities. Some people want to live in sparkling clean zero-crime Dubai wonderlands, other people want to live in grungy artsy towns with lots of local character. Some people want a nanny state protecting them from their vices, addictions, and bad decisions; other people want to be free to make their own mistakes. Some people want diverse societies with taco trucks on every corner and a local church that doubles as a mosque and gurdwara; other people want ethnostate enclaves of “thick community” where everybody is on the exact same page. Liberalism tried to offer a world where you could live life the way you wanted, but it fails when confronted with these problems; in a democracy, either the person who wants diversity or the person who wants homogeneity will end up unhappy. Worse, this is the motor which drives a hundred smaller conflicts; now you’ve got to fight for control of the public education system or the media or whatever to prevent your neighbors from being radicalized against you.
The option that many people - Robert Nozick might have been the first - proposed was a peaceful archipelago of different communities with exit rights. The people who want Dubai can found Dubai over there; the people who want Brooklyn can found Brooklyn over here; the people who want North Korea can found Pyongyang down the way.
Anyone envisioning the post-AI future needs to be thinking about the Amish. Contra the popular stereotype, the Amish aren’t anti-tech - different Amish groups accept different technologies, including solar panels, LED lights on their horse-drawn buggies, and (almost universally) medical cures. They just think that decisions about which technologies they do or don’t want should be made deliberately at the community level. I think…
