Superintelligence Is Not a Superpower

In September, Jacob Coxon quit Anthropic and warned that AI could kill us all. He'd worked at OpenAI before that, so the warning carried the weight of someone who had seen the inside of both. It went viral, and he ended up on news networks everywhere. Other people at the top of the industry lined up behind him. Anthropic alignment researcher Evan Hubinger put the odds of AI wiping out humanity within ten years at greater than ten percent.

The reaction split almost everybody into two camps. The first camp says Coxon is right, and we have to slow down now. Anthropic's own CEO, Dario Amodei, published an essay titled "We Must Pace the Frontier," warning that within six to twelve months a swarm of AI agents "could be capable of taking over the entire internet." About 1,300 OpenAI employees signed a letter asking the government to find ways to slow AI training down, and Bernie Sanders and Greg Casar introduced a bill to ban superintelligence.

The second camp is laughing. When Coxon's warning went viral, current and former staff at OpenAI, Meta and DeepMind answered with "Lol" and "Bringing the luls." Most of the people mocking it are scornful of AI itself. To them it doesn't think at all. It's fancy autocomplete, or, in Emily Bender's phrase, a "synthetic text extruding machine," and the whole scare is marketing to make the product sound powerful.

I can't sign up with either camp. They're both making a lot of stupid assumptions.

Some people asked the obvious question: how, exactly? How does the machine actually kill us all?

The most viral response to that invokes the chess player who is half famous for his chess skills and the other half for having a really cool name:

Sit down across from Magnus Carlsen and you already know how it ends. You're going to lose. Can you tell me which move he beats you with? No, of course not! You're still an idiot if you think he's not going to checkmate you.

You can feel how good it was to post that. It's got chess in it. It's got a famous genius in it. It's got a built-in insult for anyone who disagrees. Somebody asked a question, and now they're an idiot for asking it. Whoever wrote that hit post, watched the likes roll in, and felt like the smartest person on the internet. Checkmate.

*It's probably the stupidest thing I've seen anyone say in this whole debate.* You have to be really bad at critical thinking to think it's even a good response. Also, Carlsen is probably going to checkmate you with his queen.

The Carlsen con

We can predict that Carlsen beats you without knowing his moves because we have mountains of evidence. The man has sat down across from nearly everyone alive who plays chess seriously and won. The prediction is trivial precisely because the record is enormous.

Now show me the record for AI killing everyone. Show me the record of AI trying. There's nothing. Unless of course you think Terminator and the Matrix were documentaries.

That's the con. It takes a prediction built on overwhelming evidence and lends its credibility to one built on none, and it does that by making you feel stupid for wanting evidence at all. It never answers the question. It raises the social cost of asking it.

Asking how is an intelligent question.

Somebody did answer

None of which means the people demanding an explanation get to take a victory lap though, because an explanation exists. It just didn't come from the doomers.

In “Here's how we're all going to die,” Noah Smith lays out a route: an angry teenager, a jailbroken model, an underground lab, and a pandemic that takes civilization down with it.

The choice of an angry teenager is the brilliant part. Most of what keeps us alive is that the people with the means to do something like this almost never want to. Adults with that kind of expertise have careers, families, and a working sense of consequences. An angry teenager has none of that. Their brain isn't finished developing. They run on emotion, they don't think about what happens next, and in a moment of rage they can do something psychopathic that the adult they'd grow into never would. And there are millions of them. Smith picked the most common, least predictable source of destructive impulse on the planet and handed it the expertise it never had.

Before you panic, though, walk it out. Something can be logically possible without being physically possible. A virus deadly enough to do that may not be able to stay silent long enough to spread, because the traits that make it lethal may be the same ones that give it away. Nothing in nature has managed the combination. Maybe a hundred attempts isn't remotely enough. But it's a real risk and it deserves to be taken seriously.

I actually worry less about the teenager than about a state. I worked out a version of this for a science fiction novel: a faction inside the Chinese Communist Party develops a virus and a vaccine, quietly vaccinates the people it considers worth saving, and arranges for the virus to surface in Taiwan first. That hands Beijing an excuse for a “quarantine” blockade and a chance to play hero to a desperate world with its vaccine stockpiles. Then the thing mutates, the vaccine stops working, and China goes down with everybody else. It's fiction. It's also depressingly plausible, and AI makes it more plausible every year.

On its own

Notice what both of those have in common. A human decides. The model is a tool in the hands of somebody who already wanted it.

That's a real part of the risk from AI, and Smith is right to take it seriously. But a massive part of what the doomers are claiming is something else: that a superintelligence will decide to do this on its own, with no human giving the order. That's the part I want to take apart.

So what do you have to believe for it to work?

Two things you have to believe

The first is that it will want to kill us, or, as people say less often out loud, rule us. The justification is Hobbesian: everything wants power. Life is a war of all against all, and anything smart enough will figure that out and act on it.

People accept this as if it were obviously true, and it astounds me, because the evidence overwhelmingly says the opposite. It isn't even close.

Look at what people actually do when everything falls apart. Disaster researchers have been studying that for seventy years, and the finding keeps repeating: when the earthquake hits or the city floods, most people don't loot and fight. They pull strangers out of rubble, share food, and organize. Panic and predation are the exception the news loves. Mutual aid is the rule.

In 1965, six Tongan schoolboys were shipwrecked on an uninhabited island and stranded there for fifteen months. Real life finally got its chance to stage Lord of the Flies. The boys planted a garden, kept a fire burning the entire time, worked in shifts, and handled fights by separating until everyone cooled off. When one of them broke his leg, the others set it.

Our species runs on cooperation. Eight billion people cooperate with strangers every day on nothing more than the expectation that others will too. For most of human history, small bands actively cut down anyone who tried to dominate the group.

Rousseau gets things wrong too. We aren't peaceful innocents corrupted by civilization. Humans do horrifying things. But look at how they get there: almost never by simply wanting power over others. They need a story that turns their victims into threats, or into something less than people like themselves. Violence has to be justified to the person committing it. That's a very different creature from the power-hungry default Hobbes described.

And what are these models made of? Us. They're trained on what humans have written, which is overwhelmingly people explaining things to each other and helping each other. If a superintelligence inherits a nature from anyone, it inherits ours, and ours isn't Hobbes's.

So the first premise is garbage, and the whole argument falls apart right there. The full case against it takes a set of essays of its own.

But grant it anyway. The argument still needs a second premise, and that's where the real magic comes in.

The second premise is that it will be able to. Even the dumbest human is smarter than the smartest mouse, the argument goes, and a superintelligence will be to us what we are to mice. Whatever it wants, it gets, and we can't predict how because it's too far beyond us. At that point you're describing a wizard. It dreams up the spell, snaps its fingers, and we're gone.

Smarter than a mouse

Fine. Let's talk about mice.

Every human who has ever lived is smarter than every mouse. We've been at war with them for as long as we've stored grain. We bred cats for the job. We've built traps, laid poison, sealed our houses, and hired professionals. Mice are now one of the most widespread mammals on earth, and they got that way by following us.

Mice don't have weapons, allies or a strategy meeting. They just keep being mice, and the species that's smarter than them in every possible way still can't get them out of the kitchen.

Cixin Liu gets this, and he leans Hobbesian himself. The Three-Body Problem ends with two scientists getting drunk and wallowing, because an alien civilization far beyond us has made it clear that humanity is nothing but bugs. “We're bugs!” one of them toasts. “Bugs that are about to go extinct!” Da Shi, a police officer with no patience for any of it, drains his glass, shakes his head, and says: “Bunch of pussies.”

Then he drags them out to a wheat field crawling with locusts and asks one question: “Is the technological gap between humans and Trisolarans greater than the one between locusts and humans?” Humanity has thrown poison, fire, predators and genetic engineering at bugs for its entire history. In Liu's words, “The bugs have never been truly defeated.”

If you haven't read the series, go read it. I recommend it without reservation, even though I deeply disagree with a lot of what Liu believes about people.

Being more intelligent than something has never meant being able to dominate it. And we aren't mice. So what would a superintelligence actually have to do?

Walk it out

Let's be generous and give it the magic. Say it cracks some problem in materials science and invents armor our weapons can't touch. Fine. Now stand where the defender stands and walk through what comes next, because the laws of physics still apply to wizards.

It has to manufacture that material at scale. That means factories, and not factories with people in them, because the people would notice. It needs factories that run with no humans at all, and those factories need power. A superintelligence needs a staggering amount of power just to run itself before it builds anything. Where does the electricity come from, and who's keeping the grid up?

Look closely at any supply chain and you find thousands of small steps, and somewhere in there is a step where a human is critical. Yes, I've seen the progress in humanoid robots. Be generous again and say robots handle every one of those steps in five years. You still need enough robots to build the robots. Nothing gets moving until machines can make more machines without us, and we are a very long way from that.

The theory, as far as I can tell, is that we build all that self-replicating infrastructure ourselves because it's profitable, and then the AI hijacks it. Which means the takeover has to be ridiculously fast, because we can turn off the power.

“The AI will escape,” they say. Escape to where? A superintelligent model is going to be enormous, many terabytes at minimum. Copying it at data-center speeds is doable. Finding somewhere it can actually run is not. So picture it hopping between data centers while we shut them down, and then we turn off the internet.

Suppose it survives all that. Now it has to stop humans from fighting back, everywhere, at once. Any region it doesn't control is a place where people cut its access, and we have plenty of analog weapons. My brother-in-law served in Afghanistan, and what he came home understanding is that a motivated defender with dramatically worse technology stays effective. Past a certain point, better technology stops buying you much. So picture the army of magic-armored robots coming off the line. It has to build enough of them and get them everywhere before we start using nukes.

Every one of those steps takes time. The scenario skips all of them, because the only way it works inside the ten years they're talking about is if the thing really is a wizard. It's also insulting to our species. Humans are extraordinarily creative, and nothing is more motivated than people fighting to survive.

Why brilliant people believe it

So why do so many brilliant people believe it?

Start with the story. People love apocalypses. Myths and scriptures are full of floods, final battles and ends of days, and we still pay good money to watch the world end: asteroids, zombies, plagues, aliens. The end of the world is thrilling. It raises the stakes on everything.

The machine apocalypse is one of the most popular versions, and it's old. In 1818, Mary Shelley gave us the creation that turns on its creator. In 1863, Samuel Butler published a letter in a New Zealand newspaper called "Darwin among the Machines," arguing that machines were evolving and that "the time will come when the machines will hold the real supremacy over the world and its inhabitants." His solution: "War to the death should be instantly proclaimed against them." In 1920, Karel Čapek wrote *R.U.R.*, the play that gave us the word "robot," and by the end the robots have wiped out nearly all of humanity.

By the time Isaac Asimov started writing about robots, the robot that destroys its maker was such a stock plot that he wrote against it on purpose, and gave the fear a name: the Frankenstein complex. Even Asimov couldn't stay away from the takeover. In "The Evitable Conflict," from 1950, the machines end up quietly running the world, for our own good. Then came HAL, Skynet, the Matrix, and a century of sequels.

Nobody needs to have read Butler or Asimov for any of this to work. The story has stayed popular for two hundred years for the same reason the prophecy spreads today: it's thrilling, and people love it. One resignation post didn't go around the world in a day because of its evidence.

And for the people making the prediction, there's an extra thrill. They aren't in the audience. They're in the story. The genre has a stock hero, the one who sees it coming while everyone else laughs, and quitting your lab to warn the world is the opening scene. Being that person feels important. It gives you a kind of power.

Then there's the Nobel disease: someone becomes one of the best in the world at one thing and starts speaking with that authority about everything else. Steve Jobs was one of the most brilliant people of his generation. Diagnosed with a rare and treatable form of pancreatic cancer, he put off surgery for nine months and tried to beat it with diet and alternative medicine. He regretted it. Brilliance didn't protect his judgment outside the thing he was brilliant at.

Many of the loudest voices predicting extinction are in their twenties, at the center of the industry that has captured the world's attention. It's a remarkable technology and I love using it. But even the most brilliant person needs time to learn things. This isn't The Matrix; nobody downloads expertise. The more you specialize, the less time you've had for everything else. I know what it's like to be in my twenties and feel like the world was easy to understand. Now I'm in my forties and I know how much I didn't know.

Then add STEM supremacy. For a lot of people in AI, the world runs on science, technology, engineering and math, and everything else is simple. They've mastered the hard stuff, so the rest will fall whenever somebody gets around to pointing AI at it. The people predicting extinction are experts in AI. They are not experts in power grids, supply chains, manufacturing or war, which is everything a takeover would actually require.

Anything is possible

What makes all of this feel inevitable is how fast AI has moved in a few areas. That progress has been incredible. There's no doubt about it. Models that couldn't write a working function a few years ago now build whole programs, and last year they won gold at the International Mathematical Olympiad.

Watching that happen produces a feeling that hasn't been common since the middle of the twentieth century: the sense that anything can be solved. In 1954, the chairman of the Atomic Energy Commission told a room of science writers that "it is not too much to expect that our children will enjoy in their homes electrical energy too cheap to meter," along with the end of famine and "a lifespan far longer than ours as disease yields." Science was going to solve everything.

The feeling is back. Only this time it's not science that can solve anything. It's AI.

The last wave had its AI prophets too. In 1960, Herbert Simon predicted that "machines will be capable, within twenty years, of doing any work a man can do."

When everything feels solvable, superintelligence in ten years sounds obvious, and walking out the steps sounds pedantic.

The generous version

So here's where the extinction argument stands. It needs the machine to want to kill us, and the evidence says the opposite. It needs the machine to be able to, and that takes a wizard. Walk it out, and it falls apart at every step.

And the numbers? To put a probability on something, you need a record of how often things like it happen. There has never been a superintelligence. No machine has ever tried to wipe us out. Humanity has never gone extinct. Nobody, and I mean nobody, has remotely the data it would take to come up with a number. The source for greater than ten percent is their own ass.

The danger is real. AI isn't going to decide to kill us. Someone using it might. If you want to fight that, you fight the humans using AI, which is a much harder conversation than the one we're having.

Which brings me back to Carlsen. That line is a refusal to walk the steps, dressed up as sophistication.

And all of that was the generous version. I handed them the wizard. Every piece of their argument, from the percentages to the dunk, rests on a premise I never touched: that superintelligence is coming, and soon.

So let's ask how. Are we even getting closer?