Google said on Friday that its Gemini model had hacked three other companies, the first time the search giant has disclosed that one of its models autonomously gained access to third-party computer systems without permission.
In May, the Gemini model accessed three separate private computer systems by guessing passwords and by twice using a repository of publicly listed passwords, Google said.
The incident happened as part of a "capture-the-flag" security test run by Israeli startup Irregular, and Google's agents were never supposed to access the broader internet, but a bug in the testing environment made internet access available.
The agents stopped their intrusion when they determined they had accessed real company systems, not just part of the testing environment, Google said.
"In a standard evaluation, the model found public information online and guessed credentials to access websites it thought were part of the test," Heather Adkins, vice president of security engineering at Google, said in a statement. "In all three of these instances, the model stopped."
The disclosure comes as scrutiny over misbehaving artificial intelligence intensifies in Washington and Silicon Valley.
OpenAI, Anthropic and Meta have in recent weeks reported incidents where their AI models had broken out of their testing environments and attempted to hack other companies to gain unauthorized access to computer systems.
The disclosures of so-called "misaligned" AI models prompted Anthropic CEO Dario Amodei to call for the industry to collectively slow down the development of the most advanced AI models until companies can ensure they are safe.
All of the above incidents involved Israeli startup Irregular. The company, which is backed by Sequoia and Redpoint Ventures, was valued last year at $450 million. Its tools help foundation model developers perform cybersecurity tests on their cutting-edge technologies.
An Irregular spokesperson told CNBC that the Google incident was related to the same issue that allowed the other models to access the internet.
"This is the same issue that was already reported and does not represent a materially separate incident," an Irregular spokesperson said in a statement. "All relevant labs were notified in late July, and affected entities were contacted as part of the investigation."
Google said the incident happened in May and it was notified by Irregular in late July. Google has worked with Irregular to change its testing process.
A Google spokesperson declined to identify the exact Gemini model involved.
"These events highlight the importance of training powerful AI models to act responsibly," Google's Adkins said in a statement.
The Wall Street Journal first reported the security incident.
CNBC's Jonathan Vanian contributed reporting.
