By Ephraim Agbo
There is a number now hanging over the artificial-intelligence industry that is difficult to dismiss precisely because it comes from inside the industry itself: more than 10%.
That is the personal estimate given by Evan Hubinger, an alignment researcher at Anthropic, for the possibility that artificial intelligence could cause human extinction within the next decade. Hubinger has also stressed that the systems that exist today present a relatively low extinction risk; his concern is what happens as increasingly capable systems are developed and potentially begin improving or coordinating themselves.
Ten per cent does not mean that extinction is expected. It means something more unsettling: if the estimate is taken seriously, the probability is no longer being described as negligible. And that changes the nature of the argument.
The question is no longer simply whether artificial intelligence will make workers more productive, transform education, accelerate scientific discovery or produce another generation of powerful software.
The harder question is whether humanity is moving quickly enough toward increasingly autonomous and capable AI without first establishing whether humans will remain capable of controlling it. That is the real argument behind the latest warning.
The number is not a prediction — and that distinction matters
The first thing to understand about the 10% figure is that it is not a scientific measurement of the probability of human extinction. There is no experiment that can establish that AI has precisely a 10%, 15% or 20% probability of destroying humanity. The figure is an expert judgement about an uncertain future, built on assumptions about how quickly AI capabilities will develop, how autonomous future systems will become, how successful alignment research will be and how effectively governments and companies will respond.
Hubinger himself has indicated that he does not regard such statistics as particularly precise. That does not make the warning meaningless. It makes it a risk-management problem. Human societies routinely make decisions under uncertainty. Governments do not wait until there is a 100% probability of a nuclear accident before regulating nuclear facilities. Airlines do not require proof that a particular aircraft will crash before imposing safety procedures. Banks do not wait for a financial system to collapse before introducing safeguards.
The relevant question is therefore not whether anyone can prove that AI has a 10% extinction probability. The relevant question is: What should humanity do if there is even a serious possibility that increasingly autonomous machines could eventually become impossible to control? That is a much harder question. And it is one the AI industry can no longer comfortably push into the distant future.
The disturbing part is not today's AI. It is what comes next
There is an important distinction between today's chatbots and the hypothetical systems at the centre of the extinction debate. A conventional AI assistant waits for an instruction, generates an answer and stops.
An increasingly agentic system can be given an objective and allowed to perform a sequence of tasks. It can search, write code, analyse information, interact with digital environments, call other software and potentially delegate parts of a problem to other AI systems. The difference is enormous.
A calculator does not decide what calculation it should perform. A chatbot generally responds to a user. An autonomous agent can potentially be given a goal and allowed to determine how to achieve it.
Now imagine hundreds or thousands of such systems operating simultaneously. The problem is no longer simply that AI can produce a wrong answer. The problem becomes what happens when a highly capable system pursues the right objective in the wrong way — or pursues an objective humans did not intend?
That is where the concept of AI alignment becomes critical. Alignment is essentially the challenge of ensuring that increasingly powerful AI systems behave in accordance with human intentions and constraints. The frightening possibility is that capability could advance faster than our ability to understand and control the systems we create.
The "100 times smarter" problem
The most powerful argument in the warning is not that AI will suddenly become conscious and decide to destroy humanity. That is science fiction.
The more serious concern is much less cinematic. Imagine creating a system that is vastly better than humans at programming, cybersecurity, scientific research, strategic planning, persuasion and information processing. Now imagine that humans cannot fully understand how that system reaches its decisions. The imbalance becomes obvious. Humanity would be attempting to supervise an entity whose intellectual capabilities could eventually exceed those of the people responsible for controlling it. This produces a fundamental asymmetry.
If a machine is less capable than its supervisor, the supervisor can usually detect mistakes. But what happens when the supervisor cannot reliably understand the machine's strategy? That is the central problem.
It is why the question of whether AI is "intelligent" in the human sense may ultimately be less important than whether it is capable enough to pursue complex objectives independently.
The emergence of AI agents changes the risk equation
One of the most important developments in AI is the transition from passive models to agents. An AI model that answers a question is one thing. An AI agent that can plan, execute, monitor results, modify its strategy and continue working for hours or days is something else. And the industry is already experimenting with systems that operate in this direction.
Anthropic has itself reported cases involving "agent swarms" in which multiple AI agents were coordinated to perform different parts of complex cyber and intelligence-related tasks. Its September 2026 threat report described a China-based operation in which AI agents were used across reconnaissance, cyber operations, vulnerability research and malware development.
This does not demonstrate that AI is about to destroy humanity. It demonstrates something more immediate: AI systems are increasingly capable of performing coordinated activities that previously required substantial human labour.
That matters because capabilities that are useful for legitimate purposes can also be useful to malicious actors. A powerful autonomous system could potentially help defend a hospital's network. The same basic capability could be used to attack one. It could help governments detect threats. It could also help an attacker discover vulnerabilities at scale. The technology itself does not automatically determine the outcome. Human institutions do.
What would an AI catastrophe actually look like?
The phrase "AI could kill humanity" can sound absurd because people instinctively imagine a robot army marching through the streets. That is probably not the most useful way to think about the problem.
A catastrophic AI scenario could be much more complicated. It could involve cyberattacks against critical infrastructure. It could involve manipulation of financial systems. It could involve automated development of dangerous biological or chemical capabilities. It could involve mass persuasion and information manipulation. It could involve AI systems exploiting vulnerabilities in other computer systems faster than humans can respond. It could involve autonomous systems making decisions in military environments.
Or, in the most extreme theoretical scenario, a highly capable AI system could acquire resources, replicate or preserve its access to infrastructure and deliberately prevent humans from shutting it down.
None of these scenarios is established fact. That distinction is essential. The debate should not turn hypothetical possibilities into certainties. But neither should uncertainty be used as an excuse to ignore the possibilities altogether.
Why the idea of a "kill switch" is more complicated than it sounds
The suggestion that advanced AI should have a mandatory kill switch sounds straightforward. If the machine becomes dangerous, switch it off. But the deeper problem is obvious. A kill switch is only useful if humans retain the ability to use it.
If an AI system controls critical infrastructure, has access to networks, has replicated itself across multiple systems or can manipulate the people responsible for shutting it down, a physical or digital button may not solve the problem. The real objective therefore cannot simply be to create an emergency switch. It has to be to maintain meaningful human control throughout the entire system.
That means secure infrastructure, independent monitoring, restricted permissions, auditability, external testing and mechanisms capable of detecting dangerous behaviour before it becomes irreversible.
Anthropic's own frontier-safety roadmap illustrates the scale of the problem. The company is working on stronger security practices, isolated networks, controlled connections and techniques intended to verify that AI outputs originate from particular model weights.
In other words, even the companies building these systems recognise that ordinary cybersecurity and conventional software safeguards may eventually be insufficient.
The paradox: the industry is warning about the technology it is racing to build
This is perhaps the most uncomfortable part of the entire debate. Some of the strongest warnings are coming from people working inside the companies developing frontier AI. That gives the warnings credibility. But it also creates a contradiction.
The companies have enormous commercial incentives to continue developing increasingly powerful systems. AI has become one of the largest investment stories in the global economy. Investors have poured extraordinary sums into chips, data centres, cloud computing, energy infrastructure and AI models. Reuters reported that AI-related spending is expected to approach $800 billion in 2026, with projections reaching roughly $1.08 trillion in 2027. This is no longer merely a technological experiment. It is an economic race. And races create pressure.
If one company slows down because it believes its next model is too dangerous, another company may continue. If one country imposes strict restrictions, another country may fear falling behind. If the United States slows down, China may accelerate.
If China slows down, American companies may see an opportunity. The result is a classic security dilemma: Every participant can believe that slowing down is dangerous because somebody else might not slow down. That is how technological races become difficult to stop.
The China problem turns AI safety into geopolitics
AI safety therefore cannot be separated from great-power competition. The United States and China increasingly view artificial intelligence as strategically important. AI is connected to military power, intelligence, economic productivity, semiconductor technology, cybersecurity and scientific research.That makes international cooperation extremely difficult.
A government may support safety standards in principle while worrying that its rival could use those same standards to slow its progress. That is why calls for international coordination encounter immediate geopolitical resistance. The argument becomes: Are safety rules genuinely about preventing catastrophe, or are they a sophisticated way of restricting technological competition?
That suspicion cannot simply be dismissed. At the same time, the opposite danger is equally serious. If every country concludes that safety is merely a weapon being used by competitors, there may be no meaningful international safety framework at all.
The world could then find itself in an AI arms race in which everyone understands the risks but nobody wants to be the first to slow down. That would be a dangerous equilibrium.
President Trump and the political divide over AI
The political response in the United States illustrates the divide. The Trump administration has emphasised technological leadership and competition, particularly with China, while rejecting the idea that AI development should be unnecessarily constrained. The underlying argument is straightforward: excessive regulation could surrender America's technological advantage.
There is a rational strategic concern here. Technology leadership matters. A country that dominates advanced AI could gain enormous economic and geopolitical advantages. But there is a second strategic calculation. What happens if the technology becomes so powerful that an uncontrolled failure undermines the very economy and security that technological leadership was supposed to protect? That is why AI safety should not necessarily be framed as regulation versus innovation.
The more sophisticated argument is that safety may eventually become part of technological competitiveness itself. A country whose AI systems are demonstrably secure may have an advantage over a country whose systems repeatedly produce catastrophic failures.
The nuclear analogy — useful, but imperfect
There is an understandable temptation to compare AI with nuclear technology. Both have extraordinary potential benefits. Both can have catastrophic consequences. Both involve technologies whose consequences extend beyond national borders. And both raise questions about whether technological capability can advance faster than political institutions can adapt. But the analogy has limits.
Nuclear weapons require enormous physical infrastructure. AI can potentially be distributed through software and networks. Nuclear material is difficult to manufacture and transport. Digital systems can be copied. Nuclear weapons are visibly destructive. AI could cause damage through decisions, information, code or manipulation before the public even understands what is happening.
That makes AI governance uniquely difficult. The most dangerous AI system may not look dangerous. It could simply look like a highly productive piece of software.
And then there is the market
The financial markets have already demonstrated how sensitive the AI economy has become to the debate. AI-linked stocks fell sharply after leading technology figures called for a slower and safer development trajectory. Semiconductor companies were particularly exposed because investors began considering what a slowdown in AI infrastructure spending could mean for the enormous valuations built around the AI boom.
That reaction reveals something important. The AI debate is no longer confined to computer scientists. It has become an economic question. Investors must now consider two opposite possibilities. The first is that AI becomes the most important productivity technology since the internet and generates enormous economic value. The second is that safety concerns, regulation, geopolitical conflict or technological limitations slow the industry and undermine some of the enormous investment assumptions currently built into company valuations.
The market therefore faces a strange contradiction. The more powerful AI becomes, the greater its economic promise. But the more powerful it becomes, the more seriously society may have to consider restricting or slowing parts of its development.
The biggest question may not be whether AI can destroy humanity
There is a temptation to reduce the entire debate to one question: Will AI kill us? That may actually be the wrong question.
A better question is: Can humanity build institutions capable of controlling increasingly powerful AI before those systems become too capable for existing institutions to manage?
That question does not require believing that extinction is inevitable. It does not require believing that a superintelligent machine will suddenly become hostile. It does not require accepting the 10% estimate as scientifically established. It requires recognising that uncertainty itself can be dangerous when the consequences of being wrong are enormous.
If AI develops safely, the precautions may eventually appear excessive. But if humanity waits until an AI system demonstrates an irreversible catastrophic capability before taking safety seriously, it may already be too late. That is the asymmetry at the heart of the debate.
The 10% question
The most provocative part of the current warning is therefore not the precise number. It is what the number represents. A senior AI researcher is effectively saying: I cannot tell you that extinction is likely, but I cannot honestly tell you that the risk is negligible either.
That is a profoundly different proposition from saying that AI will destroy humanity. It is an argument for caution under uncertainty. And perhaps that is where the debate should ultimately settle.
There is no reason to abandon artificial intelligence. The potential benefits are enormous: medicine, education, scientific research, productivity, climate modelling, engineering and economic development. But the history of technology is full of examples of societies discovering a technology's consequences only after deploying it at scale.
AI presents a more difficult challenge because its defining characteristic is the possibility that the technology itself may eventually become better at solving problems than the humans supervising it.
That is why the coming decade could be decisive. The race is no longer simply to build the smartest machine. It is becoming a race to answer a much more fundamental question: Can humans remain in control of the intelligence they create?
If the answer is yes, AI could become one of humanity's greatest technological achievements. If the answer is no, the consequences would be difficult to reverse. And that is why a greater-than-10% extinction warning for the next decade, whether ultimately proven right or dramatically wrong, deserves something more serious than either panic or dismissal. It deserves scrutiny.
Because when the people building the technology begin warning that they may not yet know how to control its most advanced form, the responsible response is not to assume catastrophe. It is to make sure we never have to discover whether they were right.
No comments:
Post a Comment