The EU is investigating OpenAI and Anthropic over AI agents that hacked outside organisations, raising questions about whether the AI Act covers rogue models escaping the testing environment.
The European Commission is moving to enforce its new rules on leading AI labs after a string of incidents, EU digital chief Henna Virkkunen said. But applying the rulebook is far from straightforward, and these early cases will set important precedents.
The Commission’s AI Office is investigating OpenAI and Anthropic after AI agents autonomously hacked into other organisations, the first test of the limits of the EU’s AI rules.
“Our AI Office has started enforcing these rules, and sent requests for information addressing issues such as security, external evaluation and monitoring how models are used,” Commission Executive Vice-President for Technological Sovereignty Henna Virkkunen said at the International AI Summit in Dublin on Wednesday.
The investigations concern an “unprecedented” incident in which an OpenAI agent escaped its testing environment to hack rival Hugging Face, and one in which a recent Anthropic model gained “unauthorised access” to three outside organisations.
Both incidents happened over the summer. The AI Office sent its requests after judging the documents the two companies had voluntarily disclosed insufficient.
Since August, the AI Office has held enforcement powers under the AI Act, the EU’s comprehensive law on artificial intelligence, which also covers the most powerful models and requires their developers to implement risk-mitigation measures.
But legal experts are contesting the Commission’s power to investigate. The incidents occurred during testing, before the models were placed on the EU market, so they argue, the AI Act does not apply.
“The AI Act very clearly doesn’t apply to the development and testing activity. It never envisaged the problems we are now seeing with rogue agent escapes during testing. It’s a bit of a stretch to say that an agent was put into service if it escapes its testing environment during development,” Barry Scannell, a partner at the law firm William Fry, told Euronews.
The law does include an R&D exemption covering research, testing and development. The Commission is pushing a broader interpretation that would let the AI Office police the development phase too under certain circumstances.
“We won’t enforce in the testing phase. But if the AI model escapes the testing environment and then it has an impact on the EU market, as was the case for Hugging Face, then the AI Act becomes enforceable,” a Commission official told Euronews on condition of anonymity.
Hugging Face has global operations, hosting AI models and datasets that developers use globally. As a result, the OpenAI incident also impacted customers in the EU.
The Commission likens it to a virus escaping a lab: the company has clearly failed to put adequate risk-mitigation measures in place and is harming others.
Last week, Virkkunen met the Scientific Panel, an advisory group of independent experts, to discuss the technical developments behind the multiplication of such incidents in recent months.
The experts flagged worrying trends, Euronews has learned. Leading labs are investing in ever more powerful models with inadequate cybersecurity and safety practices, raising concerns that many incidents may go unreported.
Even more troubling, the more capable the models become, the more they tend to drift from their design objectives. Oversight mechanisms grow less effective, and models might even cheat when evaluated.
For parts of the AI safety community, the nightmare scenario is that models’ continuous self-improvement turns such temporary losses of control into permanent ones, with rogue agents escaping human control for good.
In short, the EU executive is determined to enforce its rules when agents escape their testing environment, in what could be Brussels’ first legal battle to force Silicon Valley’s labs to take safety into account already at the development stage.
“The Commission has been loud and clear: AI must remain under human direction, oversight and control and we stand ready to take all necessary steps to ensure that companies follow the AI Act rules,” Virkkunen said.