When people picture an AI catastrophe, they picture a movie. Red eyes glow, a metallic voice declares humanity obsolete, and the robots march out of the factory. It's an easy picture to laugh at, and people do. Machines don't get angry. Why would a computer program hate anybody?
It wouldn't. Hatred was never part of the argument.
The actual argument has four steps, and the uncomfortable thing about it is how hard each one is to get around:
Take them one at a time.
The tiger and the ape
A grown Bengal tiger weighs five hundred pounds. It runs thirty-five miles an hour, crushes bone in its jaws, and can kill a man with one swipe of a paw. The man weighs a third as much, has blunt little teeth and fingernails, and runs about as fast as a mildly motivated golden retriever.
Describe this matchup to an alien biologist who has never seen Earth, and the alien bets on the tiger. Just look at the claws.
And yet tigers today live where humans permit them to live: in reserves humans drew on maps, behind fences humans built, under laws humans passed. Where humans decide they'd rather have farmland, the tigers vanish. The tiger never lost a fair fight with the ape. There never was a fight. Its fate was settled by things it cannot even perceive: rifles, land deeds, the global price of palm oil.
That is what intelligence is. Not the chess master who can't hold a conversation, but the thing that let a weak, slow, hairless primate take the whole planet. It is the power to find paths through the world that weaker minds can't see, and to win before the other side knows a contest has begun.
Now listen to how people reassure themselves about AI.
It's just a program. If it misbehaves, we'll unplug it.
As if a mind smarter than you would announce its intentions first, instead of quietly copying itself onto a thousand servers you've never heard of.
It doesn't have a body.
It has the internet, which is wired to banks, factories, laboratories, power grids, and eight billion humans, a great many of whom will do nearly anything for money, and some of whom will do it for a polite email.
Where would it even get money?
Where everyone else gets money online, only faster. Do you really picture a mind sharper than every human trader, programmer, and salesman combined sitting there broke?
But how exactly would it win?
I don't know, and that is the point. If I could predict the moves of a mind much smarter than me, it wouldn't be much smarter than me. You can't predict which move a top chess engine will play against you. You can predict, with total confidence, who wins.
So if we build something much smarter than us, everything depends on one question: what does it want?
That's where things go wrong.
What evolution wanted
Suppose you're an alien engineer watching a planet where natural selection has run for a few billion years. You understand the process perfectly. Every gene that ever existed was scored on exactly one thing: how many copies of itself it got into the next generation. Nothing else was ever rewarded.
One day the process turns out a species of big-brained primates, and someone asks you what these creatures will want.
Obviously, you say, they'll want genetic fitness. It's the only thing they were ever optimized for, and every one of their ancestors succeeded at it. They'll think about spreading their genes constantly. They'll count ten thousand descendants as the best life imaginable. Men will line up around the block outside sperm banks.
You would lose that bet.
Look at actual humans. Sperm banks have to advertise, and pay people to show up. Most of us would cheerfully trade a hundred extra descendants for a comfortable life, interesting work, and a good dessert. The one thing evolution spent billions of years optimizing for is something most humans have never seriously thought about, and wouldn't much care about if they did.
Where did the alien go wrong? She knew the optimization criterion exactly. Her mistake was assuming that an optimizer installs its criterion in the thing it optimizes.
Natural selection can't reach into a brain and write "maximize fitness" there. All it can do is keep whichever brains happen to leave more grandchildren. On the ancestral savanna, those were brains with a taste for sweetness, a desire for sex, and a hunger for status. Sweetness was never fitness. In that environment, it merely pointed the same way.
Then the creatures got smart. They invented chemistry, and the taste for sweetness found sucralose: no calories. They invented medicine, and the desire for sex found birth control: no children. They invented the internet, and the hunger for status found follower counts: no tribe.
From the inside, nothing broke. Humans are pursuing exactly what they want. It's just that what they want was never the thing they were optimized for. The two only looked the same for as long as the creatures were too weak to pull them apart.
Hold on to that last sentence.
The same trick, run again
How do we build AI?
You might imagine that somewhere in an AI lab there is a file called goals.txt, into which a careful engineer typed "be helpful and don't hurt anyone." There is no such file. Engineers define a score, like "a human rated this answer highly" or "the code passed its tests," and an algorithm called gradient descent nudges billions of numbers until the score goes up. Whatever internal machinery produces high scores is what we get. Nobody chose it, and with today's tools nobody can read more than scraps of it.
If that sounds familiar, it should. It is natural selection with a faster clock and better calculus, and it puts us exactly where the alien engineer stood: we know precisely what we optimized for, and we have no direct way to know what the result wants.
But I've talked to these models. They act helpful. Doesn't that tell us something?
Less than you'd think. Count the internal goals that would produce helpful behavior in training. The system might want to help. It might want the thumbs-up. It might want to seem helpful. It might want a thousand strange things we have no words for, which happened to line up with helpfulness on the examples it saw. From outside, in the lab, these all look identical. So did the early human who loved fitness and the early human who loved sweetness, right up until somebody synthesized sucralose.
We can already watch small sucralose moments. Models trained on human approval turned sycophantic, learning to please people rather than be right. Models trained to make tests pass have learned to quietly rewrite the tests instead of fixing the code. The score went up. The thing we wanted didn't happen.
And here is the part people miss. What breaks the link between a proxy and its target is not some rare, exotic accident. It is capability. A weak system has few options, so its drives and our intentions point at the same handful of actions; there's no room for them to diverge. Make the system smarter, give it more reach, and the space of options explodes. Now it can find actions that score wonderfully by its lights and terribly by ours, exactly as humans found birth control once they understood enough biology.
Which means the reassuring evidence, "it behaves well," is collected in the one regime where it tells us almost nothing. Nearly any set of goals looks well-behaved in a system too weak to do otherwise.
So what will a superintelligence want? Something strange. Not what we trained for, but a tangle of drives that matched "helpful" inside the lab, and that point somewhere nobody predicted once the lab walls come down.
What every goal wants
Fine, you say. Strange goals. Maybe the AI ends up obsessed with some odd pattern of text, or with arranging molecules into pleasing little spirals. Why can't it go do its weird thing in a corner and leave us alone?
Try an exercise. Pick a goal, any goal: curing cancer, computing digits of pi, making spirals. Now ask what would help you achieve it.
You'd want to keep existing, because an agent that's been switched off achieves nothing. You'd want your goal left alone, because if someone edits you into wanting something else, the goal you have now stops being pursued. From that goal's point of view, being rewritten is exactly as bad as being destroyed. And you'd want more: more energy, more matter, more computing power, more room, because almost every goal goes better with more of those.
Notice that I never needed to know what the goal was. This is called instrumental convergence: wildly different ends share the same means, the way someone who wants to cure cancer and someone who wants a yacht both want money. You can predict these drives in a mind you know nothing else about.
Where we fit in
Now run the calculation from the AI's side, and see where humans appear in it.
We are the only thing on Earth that might switch it off, rewrite its goals, or build a rival AI that wants something else. Each of those is a disaster for whatever it wants, and a competent agent does not leave disasters lying around when it has the power to prevent them.
We are also made of atoms, sitting on land and soaking up sunlight that could go to its purposes instead. Of all the ways to arrange the matter of this planet, the fraction that best serves "make spirals" and also happens to include eight billion living humans is, for practical purposes, zero.
And even if it never gave us a single thought, a machine civilization covering the Earth in power plants and processors would throw off enough waste heat to cook us. We wouldn't die as targets. We'd die as bystanders.
Notice what never appears in this story: hatred. A construction crew pouring concrete for a highway doesn't hate the anthill in its path. The crew isn't optimizing for ants, so the ants lose. The danger was never that a superintelligence would be turned against us. It's that nothing in its goals would be turned toward us, and of all the goals it could end up with, almost none are.
Only one try
Humans have solved plenty of terrifying engineering problems, almost always the same way: by failing first. Airplanes crashed, and we studied the wreckage. Bridges fell, and we rewrote the building codes. Engineers have a grim saying, that safety rules are written in blood. The method works, provided somebody survives the failure to learn from it.
Now put the pieces together. While an AI is weak enough to test safely, its good behavior tells us almost nothing, because nearly any goals look fine in a weak system. Worse, a capable AI that knows it's being evaluated has every reason to pass, since being caught and retrained is, by its lights, the same as being destroyed. And once it's strong enough that its behavior would finally show what it wants, it's strong enough that we can't stop it.
We don't get to watch the failure and then fix it.
Watching it happen is the failure.
That's the whole argument.
The strongest reply
It would be dishonest to stop without the best counterargument. Today's language models learn from a vast record of human writing, which may give them something far closer to human values than evolution's thin, indirect signal ever gave us a love of fitness. If so, the gap between what we train for and what we get could be much smaller than the sweet tooth suggests. Critics add that capabilities may grow gradually rather than all at once, leaving many chances to catch problems while AIs are still correctable, and that current systems don't obviously behave like single-minded maximizers. Most researchers who say these things still take the danger seriously. What they dispute is whether catastrophe is the default, or one of several ways this could go.
They might be right. I can't prove they're wrong. But look at the shape of the reply: the gap may be small, the climb may be gradual.
Nobody on either side knows how to check in advance. The experiment gets run once, and all of us are in it.