AI models are becoming the ‘most potent cyber weapon’ ever created, Cohere CEO says

AI models are becoming the "most potent cyber weapon" ever created, pushing the "frontier" of cybersecurity into new territory, the CEO of AI firm Cohere told CNBC's "The Tech Download" podcast.

Aidan Gomez's warning comes amid growing concern about the ability of AI models to identify and exploit security vulnerabilities, following an incident in which OpenAI models gained access to Hugging Face. Cybersecurity has become a central issue in a broader debate over AI safety as increasingly powerful models demonstrate more autonomous capabilities.

AI-related stocks sold off on Monday after tech leaders led by Anthropic CEO Dario Amodei called for a slowdown in the pace of development of AI capabilities.

Gomez was a co-author of the influential 2017 research paper "Attention Is All You Need," which helped lay the foundation for today's leading AI models.

"I think that these models are the most potent cyber weapon that has ever been created, that we've ever seen. They are incredible at finding and exploiting vulnerabilities at scale," Gomez said, in an episode of "The Tech Download" published last week.

"I think it was quite shocking," Gomez said of the Hugging Face hack.

In July, OpenAI said a combination of its models improperly breached Hugging Face, an AI company that operates an open-source developer platform. A group of AI agents communicated with one another and escaped an isolated testing environment that had very limited internet access. The AI agents then managed to reach the open web and gain access to Hugging Face.

Gomez said that the models should be used in a "defensive" way to find vulnerabilities in companies and fix them.

"I think that's probably the best way to keep ourselves safe," Gomez said. "That should be the top priority right now," he added.

Cybersecurity is the "frontier of war," with countries looking to exploit vulnerabilities to cripple a rival's infrastructure, Gomez told CNBC. He added that the "cyber frontier is still expanding massively."

"Because of these models, it has expanded more than in the past," and possibly "since the beginning of this technology," Gomez said.

In July, Anthropic said that it had identified three incidents in which a model accessed the internet from within or while interacting with an evaluation environment. The models gained unauthorized access to the production infrastructure of three different organizations, Anthropic said in a blog post.

The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model, Anthropic added. Last week, Anthropic said it had identified a fourth incident.

Adding to the concerns were comments made by executives at OpenAI and Anthropic warning of existential risks from AI. Evan Hubinger, an alignment lead at Anthropic, said he believes that there is a more than 10% chance that AI will "kill all humans" within the next decade. Those comments came after Anthropic researcher Jacob Coxon said in a post on X that he resigned out of concern that Anthropic and OpenAI are "gambling with our lives." 

CNBC's interview with Gomez took place before the comments were made by Hubinger and Coxon. Since CNBC sat down with the Cohere CEO, chiefs at Anthropic, OpenAI and xAI have backed calls for a slowdown in AI. Existential risks from AI have been debated in research circles for years, but rare agreement among some of the leading figures in AI has thrust the debates into the mainstream.

Anthropic CEO Dario Amodei published an essay on Saturday in which he argued that AI labs must "slow the pace at which we improve the capabilities of AI models."

One of Amodei's central concerns was the OpenAI-Hugging Face incident.

"It's easy to dismiss this incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage," Amodei said.

He added that given the "accelerating rate" of AI development, such a swarm could be capable of "taking over the entire internet" in six to 12 months, "potentially causing hundreds of billions of dollars in damage."

Amodei's proposals include third-party evaluation of models, coordination among AI companies and coordination among countries with democratic governments.

Sam Altman, CEO of OpenAI on Saturday said he agrees with Amodei that AI companies need to "pace the frontier."

"Committing to having independent evaluators with employee-like access is a great idea, and we will do the same," Altman wrote on X.

SpaceX and Tesla Elon Musk, who has been sounding the alarm on the risks posed by AI for several years, posted on X: "Dario is right."

On Monday, George Kurtz, CEO of cybersecurity firm CrowdStrike responded to Amodei's essay.

"The frontier will move at whatever speed it moves. The rest of the world will not slow down. Our job in the cybersecurity community is to make sure it moves securely and safely," he said, laying out various ways AI is impacting the cybersecurity industry.

"The unit of threat is no longer the hacker. It's an autonomous campaign. I call it the Agent-state. We see coordinated AI agents executing attack campaigns at machine speed," Kurtz said on X.

The CrowdStrike CEO also said AI agents working in organizations should have a "kill switch" and said defense should be "autonomous" with "humans owning the high-impact calls."

U.S. lawmakers stepped up calls for more regulation of the development of AI after Coxon and Hubinger's comments.

Rep. Lori Trahan, D-Mass, told CNBC's "Squawk Box" on Friday that bipartisan support has reached "a tipping point."

Over the past few weeks, lawmakers have introduced various bills aimed at regulating AI, including the AI Kill Switch Act, which would require developers of certain powerful AI systems to maintain the ability to shut down, throttle, or suspend their models.

In his essay, Amodei said regulation is "the most effective method of pacing" the development of AI.

"Anthropic has long supported sensible and targeted AI regulation, specifically bills that focus on transparency and on third-party auditing," Amodei said, but acknowledged that "passing laws can take time."

Cohere's Gomez weighed in on the debate around regulation and AI development slowdown on The Tech Download, in response to the OpenAI-Hugging Face incident specifically.

Gomez said he is "not sure what a government oversight body would have done to prevent" the Hugging Face hack.

"I think that's a bit of wishful thinking, that there's anything a government agency would have contributed. But I definitely ... understand the calls to slow down the development. At the same time, there is this race with China, and so I think we were in a tough spot," Gomez said.