Artificial intelligence company Anthropic has called for a global pause on building the most powerful AI systems, as the latest models show signs they could escape human control. The San Francisco-based firm, creator of the Claude AI models, released a report on Thursday advocating for a worldwide slowdown in cutting-edge AI development, stating it would "likely be a good thing." However, the company warned that if only one entity stops, rivals would simply race ahead.
A Global Coordination Challenge
"We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," Anthropic stated. Achieving a real pause would require multiple major AI companies across countries, particularly the United States and China, to agree to stop simultaneously under verifiable rules.
This proposal may face resistance from figures like Elon Musk, whose SpaceX company—owner of AI venture xAI—is expected to debut on the stock market, potentially making him the world's first trillionaire. "Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures," Anthropic added.
Industry and Government Pushback
The company has encountered opposition from industry peers and White House officials who argue that focusing on worst-case scenarios overstates risks and serves as a strategy to slow competitors under the guise of safety. Nonetheless, the White House has acknowledged the power of Anthropic's Mythos model, which remains unreleased to the public due to its cybersecurity capabilities and is only deployed to select vetted organizations.
The proposal faces an uphill battle in Washington and Silicon Valley, where US officials and tech executives contend that any slowdown in AI development risks ceding a decisive strategic advantage to China in the defining technology race of the century. US President Donald Trump, however, mentioned discussing potential US-China cooperation on AI safety during his recent Beijing visit. Trump also signed an executive order this week mandating a 30-day preliminary review of the most powerful US AI models before their release.
Human Role Narrowing
Anthropic compared the challenge to nuclear arms control treaties but noted it would be harder to enforce since AI training is far easier to conceal than missile silos, creating enormous temptation to continue secretly. "You want the option to be able to take your foot off the gas and put your foot on the brake," said Anthropic co-founder Jack Clark on BBC Newsnight. "Right now, it's like the AI industry has a gas pedal, but it doesn't have a brake pedal."
The company plans to convene government officials, scientists, advocacy groups, and competing AI firms in the coming months to devise a workable system. Anthropic also released internal data showing that AI is already accelerating the development of AI itself, creating a feedback loop that could lead to "recursive self-improvement"—where an AI system becomes capable of teaching itself to get smarter with minimal human intervention.
"We are not there yet, and recursive self-improvement is not inevitable," the report stated, but added that it could arrive sooner than most governments and institutions are prepared for. "The evidence suggests that the human role is narrowing at each step in the AI development process," the company concluded.
