There has been no shortage of warnings from AI’s top bosses about existential threats over the years.

I attended the world’s first AI safety summit in Bletchley Park in November 2023 where the focus was on the most serious threats imaginable, posed by the most powerful AI models.

At the time, many thought this was in the realm of sci-fi and the real threats were comparatively far more mundane: job losses, cheating in exams. Three years later, many of those same voices don’t seem to be so sure.

On Saturday head of AI company Anthropic, Dario Amodei, urged the pace of development to be slowed down – the bosses of two rival AI firms, Sam Altman of OpenAI and Elon Musk said they agreed with him.

Meanwhile AI researcher Jacob Coxon – who quit Anthropic – told the BBC on Sunday staff who were developing the systems were “genuinely frightened” for the future of humanity.

They all add up to a chorus of serious voices urging a slowdown, but although it might sound like a quick fix, it is far from an easy solution.

America is widely understood to be terrified of losing the race to build the most powerful AI to China.

On Sunday President Trump made his own views clear, claiming the US was ahead of China in the AI race and declaring: “Whoever wins AI, wins.”

China isn’t renowned for wanting to come second. Everyone I speak to says that even if their company pauses, others will not and they will just get left behind. It reminds me of the height of the campaign for nuclear disarmament. The problem then too was that nobody wanted to go first.

There is also the question of how on earth a slowdown would be enforced. Who would police it? Would we be relying on the AI companies to be transparent about what they are doing, and more importantly what they are choosing not to do? That would require an epic level of trust, which arguably the tech sector has never earned.

Although Amodei in his post proposed a three-point plan that included independent monitoring of AI models as they are developed, industry-wide regulation and global regulation, some observers question how this would work in practice.

“Nobody has given a substantive explanation of what ‘slowdown’ means,” said Ed Zitron, CEO of EZ Primary Research.

“Putting a plant from METR, where one of the guys who quit Anthropic went, in every AI company? Democratic and global ‘co-ordination’? Doesn’t mean anything to me.”

Zitron said stopping training AI models would leave them vulnerable to Chinese rivals.

“Their margins might improve but their products will be captured in amber and distilled by Chinese labs. They may have to change pricing. I truly don’t know how it all works.

“This could burst the bubble. But right now we are very thin on what a ‘slowdown’ means.”

In the UK, we have heard about plans to roll out the tech more widely within the NHS to improve patient care, and how AI gave the UK economy a much-needed boost over the summer. We are all being encouraged to use it more and more at work, in education and in our personal lives.

The industry is a key part of the UK’s strategy to drive economic growth. “There is no plan B” a former government adviser told me, so could a slowdown wreck future prospects here in Britain?

The AI industry is burning through enormous amounts of money and natural resources and so far, is not creating nearly as much revenue. There are multiple reports suggesting a number of firms adopting the technology are disappointed by it.

Economists widely speculate that even the current giants are not all likely to survive and that some kind of “levelling” is coming – also known as a bursting bubble. But those firms which do make it could end up becoming the most powerful mega-corporations the world has ever seen, and that comes with its own issues.

OpenAI choosing to halt its march to the stock market could be seen as an epiphany-like moment of taking responsibility for public safety – but it could also be the move of a company which has realised it might not get the lucrative payout it needs while its entire product is perceived to be lethal.

“One of the probably the biggest challenges that we have right now is that no one can predict with certainty how this technology is going to evolve,” says Alexander Voicu from the UK AI company Synthesia.

“We know that these systems are getting more powerful, but we don’t know where and how they’re going to be used, and we haven’t figured out essentially a way of taking full advantage of their potential. My only concern in rushing to regulate now, where there’s still a lot of open questions, is that it could actually backfire.”

We also can’t deny the fact that underpinning this entire tech revolution is an ocean of investor cash.

“I’m not worried about the existential risks of AI, I’m worried about the corporate greed of the companies that are creating it,” says Sasha Luccioni, the founder of Sustainable AI.

Leading computer scientist Professor dame Wendy Hall, who has advised the UN about AI, says the current crisis is all about the companies not behaving responsibly enough.

She likens the current situation to a farmer having a bull in a field which escapes and causes destruction – and the farmer blames the bull.

“Of course it’s not the bull’s fault – it’s the farmer,” she says. Clearly, the fences weren’t robust enough, and that is exactly what she says we are seeing with AI guardrails right now.

But if those guardrails become too restrictive, could this spell the end of AI?

Some believe there is a more secretive, politically-motivated push for a rules-based clampdown in order to “regulate AI into oblivion” as Parker Thayer, an investigative researcher at the Conservative-leaning Capital Research Centre think-tank, put it on X this week.

His post was viewed nearly eight million times. It is an extreme and unproven view but it shows that not everybody is on board with the idea of regulation saving the day.

Whatever the reality, the storm currently engulfing the AI industry could already have caused reputational damage to the firms currently in pole position forever.

As Professor Hall puts it: “Would you invest in a company that says it’s going to bring about human extinction?”