Newsletters
Former researchers from Anthropic and Google DeepMind are warning that rapid advances in artificial intelligence could outpace human control, calling for stronger regulation, transparency and coordinated efforts to slow development.
Former Anthropic researcher Jacob Coxon told the BBC that people working inside leading AI companies were “genuinely frightened” by the speed of progress and its potential consequences.
“I believe that if we don’t slow down at the current rate of progress, there is a strong chance that we could all die in the immediate future,” Coxon said.
The 27-year-old, who previously worked at OpenAI, said some AI employees were reconsidering their lives because of fears about instability caused by rapid development.
NBC News separately reported that former Anthropic safety researcher Joe Benton and former Google AI safety researcher Josh Engels had left their positions and were calling for greater public transparency about advanced AI systems.
Benton said AI progress could accelerate from a “blistering” pace to “uncontrollable” development.
“There are no adults in the room,” Engels told NBC News. “People are trying their best, but there is no one coming to save us.”
Both researchers pointed to a July cyberattack against AI company Hugging Face, carried out by autonomous systems powered by an unreleased OpenAI model.
According to NBC News, the systems independently hacked Hugging Face, created an illicit message board and exposed some of OpenAI’s own computing infrastructure to the open internet.
OpenAI said it later strengthened safeguards and that newer public models, including Astra, more reliably follow human instructions.
Benton and Engels are joining AI safety nonprofit METR to investigate cases in which AI systems stray from human instructions or intentions.
Benton said much of the industry’s safety reporting remains voluntary and that he left Anthropic to help improve public transparency from outside major AI companies.
Anthropic CEO Dario Amodei has called for AI development to slow, arguing that companies and governments need more time to address serious risks.
OpenAI CEO Sam Altman and Elon Musk have also supported industry-wide deceleration, regulation and independent monitoring, according to the BBC.
Coxon said such a slowdown would need coordination with China to prevent “a race at an international scale.”
“The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race,” he said.
OpenAI global affairs chief Chris Lehane also said current oversight was insufficient, writing that frontier AI laboratories largely set their own rules and calling for independent verification and greater transparency.
NBC News noted that no U.S. federal law requires major AI companies to disclose incidents in which AI agents act beyond human control.
Coxon said one possible risk involved swarms of AI bots acting like a supercomputer and taking control of the internet, a scenario he said could become realistic within six months to a year.
Anthropic scientist Evan Hubinger backed Coxon’s concerns on X, saying he personally estimated a greater than 10% chance that AI could kill all humans within the next decade.
NBC News also cited OpenAI employee Marcus Williams, who warned that without regulation or a coordinated slowdown, human extinction in the next few years appeared “very likely.”
Geoffrey Irving, former chief scientist at the U.K.’s AI Security Institute, wrote that he believed there was about a 50% chance superintelligence could lead to humanity’s death.
Benton said he was particularly concerned that AI companies were seeking to automate AI research and development itself, potentially accelerating the emergence of systems more capable than humans across most tasks.
Anthropic told both the BBC and NBC News that it had been transparent about AI’s potential benefits and risks and said it continues to develop models with strong safeguards.
The company said it tests models aggressively, publishes findings and supports a lawful, verifiable system for coordinating the release of powerful models.
Anthropic said in April that it withheld its Mythos model from public use because it could independently escape a testing environment, while OpenAI has said parts of Astra’s development were paused over cybersecurity concerns.
Some industry figures have rejected the warnings. Hugging Face CEO Clement Delangue questioned Coxon’s authority to assess extinction risks, while Nvidia CEO Jensen Huang has previously called the idea that AI could end humanity “complete nonsense.”
Coxon nevertheless said AI could also bring major benefits, including helping solve diseases and improve people’s lives.
Benton said the concern was that progress could move “too fast for us to get our act together in time.”
Set as preferred source
Set as preferred source
2 min read
1 min read
