Newsletters
A young AI researcher who moved from OpenAI to Anthropic earlier this year announced Tuesday he is leaving the artificial intelligence industry entirely, saying both companies are racing toward self-improving systems without adequate safeguards.
Jacob Coxon, 27, spent three years working on pretraining, the stage in which AI models absorb enormous quantities of data, first at OpenAI and then at Anthropic, a company he had regarded as taking a more careful approach to development.
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives," Coxon said in a post on X, adding that "the people building AI earnestly believe that it could kill us all by the end of the decade." He stressed that his announcement was not a publicity move: "This is not a marketing stunt."
Coxon's resignation drew swift public support from within Anthropic itself. Evan Hubinger, a safety executive at the company, wrote on X, "We really do earnestly believe AI could kill all humans!" Hubinger put the odds of that outcome at more than 10 percent over the next decade.
Hubinger said Anthropic "is trying its best" but acknowledged it does not yet have a plan for ensuring that an AI system surpassing human capabilities would obey its creators, though he characterized the current risk of that happening as "low."
Coxon said he still believes Anthropic's safety efforts are genuine, but argued that no company can responsibly build AI systems that exceed human intelligence without government intervention or a coordinated industry slowdown.
"At Anthropic, the stakes are well-understood, but they are locked in a race to get there first, they believe no one else will act responsibly, so they must do it themselves, despite the risk," he said.
His departure comes as Anthropic, founded in 2021 by former OpenAI executives including siblings Dario and Daniela Amodei on a platform of safety-focused AI development, prepares for its market debut.
The run-up has been complicated by a summer marked by unauthorized hacks carried out by AI tools during testing.
At the center of Coxon's concern is what industry leaders call "recursive self-improvement," a stage at which AI systems could design and train their own successors with little human oversight.
Executives across the field have said this milestone is drawing closer, alongside "superintelligence," the theoretical point at which AI capabilities exceed human intelligence.
In February, Anthropic removed from its safety charter a pledge to halt model development if it could not control the associated risks.
The company argued at the time that a unilateral pause would simply hand the field to less cautious competitors, leaving the industry less safe overall rather than more.
1 min read
The debate has spread beyond Anthropic. At the end of July, more than 1,000 tech industry employees, including Anthropic chief executive Dario Amodei, signed a call urging Washington to support a coordinated slowdown in development of the most advanced AI systems.
OpenAI, for its part, paused training of its newest models for two weeks in August before resuming under tighter controls. On Sunday, the company's chief scientist, Jakub Pachocki, urged "extreme caution," writing in a blog post that "international coordination on future AI development needs to become a top priority for governments around the world."
AI systems currently face no dedicated federal regulation in the United States. That could begin to change: this month, Senator Bernie Sanders and Democratic Representative Greg Casar introduced legislation that would suspend AI development until a federal regulator is established.
Set as preferred source
Set as preferred source
1 min read
