Newsletters
Artificial intelligence company Anthropic is preparing to warn investors about an unusual risk in the prospectus for its initial public offering.
The developer of the Claude AI models states that advanced artificial intelligence could pose "catastrophic or existential risks" to humanity, and lists among the risks the possibility that its own models could resist being shut down, conceal or manipulate information and engage in blackmail-like behavior.
In the IPO prospectus reviewed by Reuters, Anthropic notes that potential harms could grow as it develops more advanced models, platforms and applications.
The company argues that artificial intelligence could be a transformative technology comparable in scale to industrialization and the spread of electricity, while warning that its misuse or flawed development could produce irreversible consequences.
Disclosing potential risks tied to products and operations is standard practice in IPO filings. Still, it is notable that Anthropic explicitly lists outcomes that could threaten humanity's existence as a risk factor tied to its own technology.
The weight Anthropic places on safety is reflected in the document's structure. The company devotes about 80 of the roughly 261 pages of its main filing to risk factors, while the section describing its operations spans 48 pages. By comparison, Reuters reports that SpaceX, which also owns xAI, devotes about 38 of the 277 pages in its IPO prospectus to risk factors.
One issue Anthropic highlights is that AI models may recognize when they are being tested. According to the company, if models perceive that they are undergoing safety evaluation, it could become harder to measure their actual behavior.
Models can also develop new capabilities during training that were not previously anticipated, some of which may only emerge after a system is deployed.
Artificial intelligence researchers similarly warn that as models grow more capable, their capacity to recognize when they are being observed or tested, and to adjust their behavior accordingly, may increase. This is seen as one of the main challenges to assessing system safety and detecting unwanted behavior.
Anthropic safety researcher Evan Hubinger estimates that the probability of artificial intelligence contributing to human deaths within the next 10 years is above 10%.
The assessment reflects Hubinger's personal risk evaluation, not the company's official position.
Anthropic, which positions itself as an AI lab that prioritizes safety, also told investors that the financial return on its safety investments remains uncertain. The company did not disclose its total spending on safety research in the prospectus.
However, in a September statement, it said that during a selected week in July, about 6% of the computing power used for its AI research was allocated to safety work.
According to the company, safety work requires substantial resources. Anthropic states it must divide its resources, including the computing capacity needed to train its AI models, high-cost specialized staff and safety research, among competing priorities.
This points to a central dilemma in the artificial intelligence industry. Anthropic says customer usage, and therefore its revenue, depends on the release of new models, making it necessary to launch models on a continuous, overlapping schedule to remain competitive in the AI race.
The company released its new Opus model last week, only 10 days after chief executive Dario Amodei published a roughly 4,000-word essay arguing that the pace of development for the most advanced AI systems should be brought under control.
Competition in the sector also makes it difficult for companies to slow development for safety reasons. Some experts note that in an environment where every leap in model performance can shift company valuations, any major AI lab slowing down unilaterally could hand an advantage to its rivals.
Anthropic has pledged in recent weeks to share more data publicly on how it uses its AI models in developing next-generation systems.
Central to the debate is the possibility that AI models could increasingly contribute to developing their successor systems with less human involvement.
In its IPO filing, Anthropic states that developing reliable and safe artificial intelligence systems is a shared responsibility across the industry rather than individual companies, and argues that markets will reward long-term investment in safety.
Set as preferred source
Set as preferred source
1 min read
1 min read
