Newsletters
Anthropic, the San Francisco company that created the Claude AI models, is asking for a global slowdown in developing the most advanced AI systems. The company warns that new models are starting to show signs that they might move beyond human control.
In a report released Thursday, the company said a global pause on advanced AI development would "likely be a good thing." However, they noted that if only one company stopped, others would keep moving forward without limits.
"We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advances of the technology," the company said.
Anthropic said that for a pause to work, several major AI companies in different countries, especially in the U.S. and China, would need to agree simultaneously to stop development under a system that everyone can verify.
The company warned that without a way for countries to work together, companies and governments would have to make important safety decisions amid intense competition and political pressure.
Some people in the industry and White House officials have pushed back against the proposal. They argue that Anthropic is focusing too much on worst-case scenarios, exaggerating the real risks, and trying to slow down competitors.
Anthropic admitted that its proposal faces strong resistance in Washington and Silicon Valley. Many officials and executives there say that slowing down AI development could mean losing ground to China in what they see as a key technology race.
U.S. President Donald Trump said he talked about working with China on AI safety during a recent trip to Beijing. He also signed an executive order this week that requires a 30-day government review of the most powerful U.S. AI models before they are released to the public.
Anthropic compared the challenge of coordinating an AI pause to nuclear arms control, but said it would be much harder to manage. AI training is much easier to hide than a missile silo, and there would be strong reasons for companies to keep developing AI in secret.
The company said it plans to bring together government officials, scientists, advocacy groups, and other AI companies over the next few months to discuss how to establish a system for collaboration.
This call comes as Anthropic's internal data show that AI is already accelerating its own development. The company warned that this feedback loop could eventually lead to "recursive self-improvement," in which an AI system could improve its own intelligence with little help from humans.
"We are not there yet, and recursive self-improvement is not inevitable," the report said, while cautioning that it could arrive sooner than most governments and institutions are prepared for.
"The evidence suggests that the human role is narrowing at each step in the AI development process," Anthropic said.
Set as preferred source
Set as preferred source
1 min read
1 min read
