AI is already writing essays, coding programs and answering questions in seconds. Some of the people creating it now argue that the same technology could eventually become much harder to control.
A few years ago, the idea of artificial intelligence wiping out humanity sounded like something from a bad sci-fi movie. Robots turn evil, machines take over the world, humans lose. It was entertaining, but not something most people actually worried about.
That has started to change.
Some of the warnings are now coming from the people who understand AI best. Earlier this month, Anthropic researcher Jacob Coxon resigned from the company and warned that the race to develop more powerful AI could become dangerous enough to threaten humanity. Evan Hubinger, another Anthropic researcher whose job focuses on controlling advanced AI systems, publicly estimated the chance of AI causing human extinction within the next decade at more than 10%.
Those are obviously extreme predictions. But when researchers at the companies actually building these systems are talking about extinction as a serious possibility, it is worth understanding what they mean.
The first thing to know is that almost nobody is seriously predicting an army of Terminator-style robots suddenly deciding that humans are annoying. The concern is stranger than that.
Imagine giving an extremely intelligent AI one goal. Maybe you tell it to run a company as efficiently as possible. A normal computer program follows a limited set of instructions, but a much more capable AI could potentially make plans, use other software, communicate with people and find new ways to accomplish what it was asked to do.
The problem comes if the goal we give it does not perfectly match what we actually want.
There is a famous example in AI safety involving paper clips. Imagine an incredibly powerful AI whose only objective is to manufacture as many paper clips as possible. It does not hate humans. It does not feel anything at all. But if it becomes powerful enough, almost everything humans value — land, factories, resources and eventually humans themselves — could become obstacles to producing more paper clips.
Obviously, nobody is planning to build the world’s most powerful paper-clip machine. The example is meant to show something more basic: being intelligent does not automatically mean understanding what humans care about.
This is known as the “alignment problem.” Developers can tell an AI what they want, but making sure a highly capable system continues to behave according to human intentions is much harder. Researchers are especially concerned about future systems that could operate independently for long periods of time or improve their own abilities. The Wall Street Journal reports that AI companies are actively researching ways to control these kinds of systems, but there is still no proven solution for controlling a hypothetical AI that becomes significantly smarter than humans.
There is also a much simpler danger: humans.
AI does not have to turn against us to cause serious damage. A powerful system could be deliberately used by someone to carry out cyberattacks, spread highly convincing false information or help develop dangerous biological weapons. Anthropic CEO Dario Amodei has recently warned about both cyber and biological risks from increasingly capable AI systems.
In that situation, the problem is not an evil computer. It is a person with access to an extremely powerful tool.
That difference matters because some AI risks already exist in smaller forms today. AI-generated scams and misinformation are real. Computer systems can already help people write malicious code. The argument from AI safety researchers is that these problems could become much more serious as the technology improves.
Concern about this is not limited to a few people on the internet. In 2023, more than 700 AI researchers and public figures signed a statement from the Center for AI Safety saying that reducing the risk of extinction from AI should be treated as a global priority. Signers included Geoffrey Hinton, Yoshua Bengio, Google DeepMind CEO Demis Hassabis, OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei.
Still, none of this means an AI apocalypse is definitely coming.
Predictions about future technology are notoriously unreliable, and there is a huge difference between today’s chatbots and a machine capable of overpowering humanity. Even among AI researchers, there is major disagreement about how likely these scenarios really are. Critics also argue that talking constantly about hypothetical human extinction can distract from problems AI is causing right now, including cheating in schools, job disruption, deepfakes, scams and the huge amount of power being handed to a small number of technology companies.
That criticism is fair. A 10% estimate of extinction risk is not the same thing as scientific proof that there is a 10% chance everyone dies. Nobody has run this experiment before. These numbers are ultimately educated guesses about technology that does not exist yet. But there is also a strange contradiction at the center of the debate: if some AI developers genuinely believe the technology could become catastrophically dangerous, why are they still building it?
Part of the answer is competition. Companies do not want to fall behind their rivals, and countries do not want other countries to gain control of the world’s most powerful technology first. Developers also argue that AI could have enormous benefits, from scientific discoveries to medicine, and that the answer is to make it safer rather than stop developing it completely. The result is basically a race where several of the competitors are saying the finish line might be dangerous, while also worrying about what happens if someone else gets there first.
That may be the most important part of the whole debate.
We do not need to decide whether AI will save humanity or destroy it. Both predictions are mostly speculation. What we do know is that AI systems are becoming more capable, enormous amounts of money are being spent developing them, and even some of the people leading that development think there are serious risks. Maybe the doomsday predictions will eventually look ridiculous—and hopefully they do. But if the people building one of the most powerful technologies in history are warning that they are not completely sure how we will control it, asking questions before it becomes even more powerful seems pretty reasonable.
The Blue and White 