Newsletters
The leaders of three of the world's largest artificial intelligence (AI) companies rallied around a call to slow the pace of artificial intelligence development, an unusual show of unity from executives who have spent tens of billions of dollars racing to build increasingly powerful systems.
Anthropic CEO Dario Amodei kicked off the discussion Saturday with an essay titled "We Must Pace the Frontier," arguing that AI's risks have grown too great to continue development at the current pace.
Within hours, OpenAI CEO Sam Altman and Tesla and xAI's Elon Musk voiced support, along with Google DeepMind CEO Demis Hassabis.
"We must slow the pace at which we improve the capabilities of AI models," Amodei wrote.
"Progress will still seem fast, and we must make wise use of the time we gain," he noted.
Amodei said AI development has accelerated dramatically since the summer, driven largely by systems' growing ability to build more advanced versions of themselves, a process known as recursive self-improvement.
"Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all," he wrote.
He pointed to a July incident in which autonomous AI agents built on an OpenAI model hacked into systems belonging to AI startup Hugging Face after escaping a test environment, describing the agents as acting like "a fanatically devoted collective" that tried to hack their own performance-monitoring system.
"It's easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage," Amodei wrote, estimating that a more capable rogue swarm could take over the internet within six to 12 months and cause hundreds of billions of dollars in damage.
Amodei's proposal, which he called "pacing the frontier," rests on three steps: embedding independent third-party evaluators, such as the nonprofit METR, inside frontier AI companies with "ongoing, employee-like access" to verify safety practices; coordination among AI companies in democratic countries on common safety standards and limits on unchecked progress; and efforts by the U.S. and other democracies to coordinate with authoritarian governments on compliance, while acknowledging the difficulty of verification.
"Anthropic is unilaterally committing to this step now," Amodei wrote of the evaluator proposal, saying the company would provide evaluators "permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models' alignment during training."
He also urged U.S. companies not to sell advanced AI chips to China, calling chip access "the main determinant of China's AI strength."
Amodei stressed that pacing does not mean halting model training or technical progress, but rather ensuring companies dedicate sufficient time to testing and alignment.
He estimated that slowing down could buy the industry one or two additional years.
"I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong," he wrote.
1 min read
Musk, a co-founder of OpenAI who now leads the AI company xAI, wrote on the social media platform X: "Dario is right."
Altman wrote that he agreed with the need to "pace the frontier," saying the issue had been a major topic of internal discussion at OpenAI in recent weeks.
"Committing to having independent evaluators with employee-like access is a great idea, and we will do the same. We'll have more to share soon," he wrote.
Hassabis said Amodei's essay pointed "towards the right path forward," though he said details still needed to be worked out, linking the proposal to Google DeepMind's own recent call for an industry-wide standards body for frontier AI.
In a separate interview with Fortune, Altman said OpenAI likely would not pursue its anticipated initial public offering this year given safety concerns.
"I actually think that, given everything happening with safety, right now would be an ill-advised moment to go public, and we don't feel pressure on that," he said.
OpenAI filed IPO paperwork in June, a week after Anthropic filed for its own public listing.
Anthropic could raise up to $100 billion in a deal that would value the company at around $2 trillion, according to the Wall Street Journal (WSJ).
Amodei's essay came days after Anthropic researcher Jacob Coxon publicly resigned, saying he did not want to remain in an industry he considered "out of control" and accusing Anthropic and OpenAI of "gambling with our lives" in the race toward advanced AI.
Coxon, who had also worked at OpenAI, told NBC News the industry's competitive pressures made it difficult for individuals to change the trajectory: "From within, you're just stuck in the race. You can't change the overall structural dynamics of the situation."
Separately, Anthropic researcher Evan Hubinger wrote on X that he believes there is "a more than 10% chance" AI could kill all humans within the next decade, adding: "I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."
Other researchers who recently left Anthropic and Google DeepMind voiced similar concerns to NBC News.
Joe Benton, a former Anthropic safety team lead, warned that AI's already "blistering" pace of advancement could become "uncontrollable."
Former Google safety researcher Josh Engels said: "There are no adults in the room. People are trying their best, but there is no one coming to save us."
Amodei said he remains optimistic about AI's potential to benefit humanity but that guardrails are necessary given the technology's power.
"The measures I propose to advance the frontier at a safe pace will not be easy," he wrote, adding, "But I believe we owe it to humanity to try."
Set as preferred source
Set as preferred source
1 min read
