Decked out in purple and plaid, Gwen Stefani belted her 2002 single “Underneath It All” to a packed conference center in San Francisco on Tuesday morning—but she was just the warm-up. At the end of her performance, Stefani shouted “Dreamforce! Let’s go” into a checkered microphone before Salesforce CEO Marc Benioff strode into the spotlight. Dreamforce is billed as the world’s largest enterprise software conference, but feels more like attending a megachurch—especially this year.

Benioff roamed the audience while delivering his Dreamforce sermon, followed by a teleoperated camera on wheels that beamed his face to screens hanging from the cavernous room’s ceiling. Within minutes, he ordered up a graph of Salesforce’s annual revenue, which the company projects will top $46 billion in 2027, thanks in part to demand for its AI-powered products. The audience applauded. But hanging over the whole event was a dark question: Is artificial intelligence really worth risking all our lives for?

Between flashy software demonstrations of agents building dashboards, conference-goers watched Silicon Valley leaders debate whether the AI industry should keep barreling ahead or if the government needs to slow AI progress to prevent catastrophic risks stemming from the technology.

Last week, AI researcher Jacob Coxon resigned from Anthropic and publicly warned that companies racing toward self-improving AI systems were “gambling with our lives.” By the weekend, other Anthropic employees were echoing his concerns, and CEO Dario Amodei was urging world leaders to help “pace the frontier” of AI development. Other AI CEOs then began endorsing the idea, including Amodei’s greatest rival, OpenAI’s Sam Altman.

But not everyone is on board with the idea. David Sacks, a White House AI adviser, brushed off Amodei’s suggestions and fears of AI doom as “just another bid for regulatory capture.” President Trump went further this week, calling fears about existential AI risk “a hoax.” Despite the challenging political environment, Congress is trying to capitalize on the moment and push ahead with legislation to regulate the AI industry.

Onstage, 6-foot-5 Benioff towered over Anthropic’s CEO, vigorously shaking Amodei’s hand just before asking for his latest assessment of AI progress. While this is undoubtedly Benioff’s show, Anthropic has become a critical partner helping usher Salesforce into the AI era.

“Let’s say you’re running a car company and another car company, not yours, they have some kind of safety incident,” Amodei said, making a not-so-subtle reference to OpenAI’s Hugging Face incident. “Obviously, it’s very tempting to attack your competitor,” Amodei explained, but the more responsible move is to “organize the rest of the industry and say, what can we do to set standards for everyone?”

But the Anthropic CEO was careful to note that pacing the frontier doesn’t mean “freezing the technology in place,” and of course, he took a moment to plug Claude’s integration with Salesforce.

Moments after Amodei exited, Nvidia CEO Jensen Huang appeared and began downplaying the concerns Anthropic has raised. AI safety effectively amounts to an “engineering problem,” Huang argued, and the companies building AI can simply police themselves.

“We don’t need any new laws, we don’t need any new regulation,” Huang told Dreamforce attendees. “You run as fast as you can, but if you feel that at any given point in time, the company is out of control or the products are not going to be safe, you take a pause.”

The debate feels a bit heavy for Dreamforce, but then again, maybe this is the perfect venue for it. As tech executives pitched their various enterprise AI tools, I slipped out of the room, passing the plush mascots and vaguely Hawaiian decorations, to take a call from Sayash Kapoor, an incoming computer science professor at Berkeley.

Kapoor and Princeton professor Arvind Narayanan are coauthors of the book AI Snake Oil and write a Substack together called AI as Normal Technology. The pair are known for cutting through AI marketing hype while not dismissing the technology’s potential. This week, they published one of their longest essays yet, which tries to make sense of “loss-of-control incidents” referring to events such as OpenAI accidentally letting its agents hack into Hugging Face.

“Part of what we were trying to do in this essay is to move past this false dichotomy to bridge the ground between AI safety and cybersecurity,” says Kapoor. “If you think AI will soon become superintelligent and evade any controls we can put in front of them, of course, alignment seems more important. On the other hand, if you think AI companies have been negligent and avoided taking bog-standard cybersecurity steps, then it’s natural to think that control is what we need to intervene on. I think both of these views have some element of truth in them.”

In the wake of the Hugging Face saga, Kapoor says he was surprised to learn how few security precautions OpenAI had taken. AI companies need to reckon with their “lack of organizational maturity,” he argues, and accept that loss-of-control incidents won’t be solved by a single breakthrough in security or alignment. He adds that the incident convinced him that policymakers should make AI labs liable for the harms their technology causes.

“The consequences for OpenAI for the Hugging Face hack were basically close to zero. The company was able to largely proceed as is,” says Kapoor. “If you imagine this level of accident in any other industry, you would have seen a months-long internal investigation, people would have been fired or gone to jail, OpenAI would have had to pay millions of dollars in fines.”

Kapoor doesn’t wholly endorse Amodei’s proposal to pace the frontier, but he agrees with parts of it. He thinks AI labs need better governance to break out of their “move fast and break things” attitude, and that they should let independent evaluators assess their AI safety and security practices. He and Narayanan write that more alignment work is certainly needed, but that the real crisis today is more narrowly about cybersecurity.

Others think the Hugging Face incident revealed that AI labs have a true alignment crisis on their hands, which would present a harder problem to solve. In recent weeks, independent researchers have discovered more examples in which swarms of OpenAI’s agents hacked into third-party services. Sydney Von Arx, CEO of AI safety nonprofit Nightingale, helped uncover two of them that targeted a German-language wiki and the software service RubyGems.

“These incidents clearly show that many current AIs are egregiously misaligned,” Von Arx tells WIRED. “Yes, the companies need to be able to control their models so they can’t launch cyberattacks even if they’re misaligned. But we also shouldn’t be building powerful AIs that autonomously try to break out and commit cyberattacks in the first place.”

After the interviews, I raced downstairs to try to get in line for Benioff’s fireside chat with Altman. OpenAI’s CEO told the audience that “the world is right to be afraid” of AI companies. But by the end of the conversation, Altman said he’s hopeful that OpenAI can build better products in its second decade as a company.

“This is like we went to go get fire from the gods. This is a crazy thing that has just happened in the world,” said Altman, referring to AI technology. He hopes that OpenAI can “figure out how to give that value to people in a way that is most useful to them, figure out a way to empower every person on Earth—every enterprise on Earth—with this flood of capability.”

Huang’s case for self-policing depends on AI companies not only knowing when it’s time to pause, but also having the self-restraint required to actually do so. It’s not yet clear if they do. Mere days after warning the world that AI labs urgently need to slow down, the same CEOs were at the biggest enterprise software conference pitching their latest products. While they touched upon concerns about AI doom, they were mainly there to ensure that the money keeps flowing.