King Charles has warned AI executives of the “existential dangers” posed by the technology falling into the wrong hands.

He made the comments at a summit he had convened at Dumfries House in Scotland to discuss how the tech can be used to benefit society.

Those gathered at the meeting also included Kanishka Narayan, the UK’s AI Minister, the Pope’s advisor, and participants from AI giants such as Nvidia, OpenAI and Anthropic.

It comes amid a series of recent warnings from people inside the industry about the rapidly evolving capabilities of advanced or “frontier” AI models and the regulation needed to keep them in check.

“Those who have created these technologies are now increasingly warning that AI risks developing darker capacities – perhaps even to take life,” The King told the executives.

He said there seemed to be an “urgency” in “adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways”.

“The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world,” he told the group.

The summit has been called to consider whether a “shared set of principles” can be developed to “help guide the future application” of AI.

Since August, a wave of discussion about AI’s potential power and how to control and regulate it has dominated blog posts and social media, emphasised by several dramatic statements from those in the industry.

The debate has also been met with criticism from sceptics who say the focus on AI’s hypothetical future risks distracts from more immediate concerns around the technology, such as its environmental impact.

In July, OpenAI announced what it called an “unprecedented” incident in which its advanced AI escaped test limits and hacked into the tech platform Hugging Face.

After several other incidents, in September, a resignation post from Anthropic researcher Jacob Coxon went viral when he cited concerns AI could wipe out humanity.

He later told the BBC the people working on the technology were “genuinely frightened” about the speed of its advancements and what it could mean for humanity.

In response, Anthropic scientist Evan Hubinger wrote he personally thought the possibility of human extinction from AI was a greater than 10% chance.

The AI companies’ top bosses have all weighed in with their thoughts following the claims, with OpenAI’s Sam Altman saying people were right to be concerned, but adding they should trust AI firms to keep the tech safe.

Anthropic’s Jack Clark said a so-called “kill switch” may be needed to protect humanity, while Microsoft’s Mustafa Suleyman criticised Anthropic, claiming they were training their AI to believe it was human, which could make it “impossible” to control.

Critics, meanwhile, argue that AI firms cannot be trusted to regulate themselves.

Some, such as left-wing Senator Bernie Sanders and Steve Bannon, a former adviser of US President Donald Trump, are calling for stronger government oversight.

Others, including Donald Trump himself, have dismissed concerns about AI safety as overblown – with the president calling fears about it a “hoax”.

Sign up for our Tech Decoded newsletter to follow the world’s top tech stories and trends. Outside the UK? Sign up here.