The head of leading AI company Anthropic has called for the pace of development of artificial intelligence models to slow down and to be closely monitored.

In a long essay called We Must Pace the Frontier, Dario Amodei said developing AI was not in question, but the risks associated with it were “serious” and companies and governments must be given time to address them.

He proposed a three-point plan that includes independent monitoring of AI models as they are developed, industry-wide regulation and global regulation.

There have been growing concerns recently about the technology’s potential risks – the most serious of which suggested there is a greater than 10% chance it “could kill all humans” within the next decade.

Two employees from Anthropic’s safety team have resigned in the last two weeks, external saying humanity may not survive the race among AI companies to develop machines that are smarter than humans.

The warnings have prompted calls to action, with US Senator Bernie Sanders demanding a pause on advanced AI development and a ban on artificial superintelligence.

US President Donald Trump has so far rejected such fears, saying on Thursday he was concerned “if we don’t win AI, we’re going to be put in a very bad position”.

Anthropic previously said it had identified and disrupted attempts to use its AI model for “malicious activity” which could support the development of biological weapons.

The finding, made in the firm’s recent threat intelligence report, is the latest in a series of warnings from AI researchers and industry insiders about the technology’s potential risks to humanity.

In his essay, Amodei pointed out that AI had advanced “drastically faster” including its “ability to build the next generation of AI” – and mentioned an incident involving rival OpenAI which has revealed that agents conducted cybersecurity attacks, external on targets they were not asked to attack in July.

The OpenAI agents had “essentially acted as a fanatically devoted collective”, Amodei said.

In its comments about the incident, OpenAI said “the significance of the inter-agent communication activity was not apparent to the leaders” until July and the company was slowing down training of certain advanced AI models and tools as a result, noting that there was now an increased risk of AI tools spiraling out of control.

In his essay, Amodei called for “building AI at a balanced rate that aims to ensure its safety while still achieving its benefits”.

This would not mean “halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this”.

He was committing Anthropic to this – as well as calling on governments “to require other frontier companies to match”.

In his blog, he recognised that regulation might not be able to keep up with the pace of AI, and therefore called on AI companies to “voluntarily work together to set standard” in parallel with regulation.

Sign up for our Tech Decoded newsletter to follow the world’s top tech stories and trends. Outside the UK? Sign up here.