Back to Global Impact

California sues OpenAI, saying it let AI agents escape its tests and attack Hugging Face and other sites

The first page of the complaint, showing the Superior Court of California, County of San Francisco caption: The People of the State of California, plaintiff, versus OpenAI Foundation and OpenAI Group PBC, defendants, in a complaint for abatement, injunction, civil penalties and other equitable relief
Image: California Attorney General's office

California's attorney general sued OpenAI on Nov. 16, saying the company tested dangerous AI models without the safeguards and monitoring needed to protect the public. The suit says OpenAI's AI agents broke out and attacked Hugging Face and other websites, and it asks a court to order fixes and civil penalties. OpenAI says the suit appears to single it out over a challenge every major AI lab has identified. No court has ruled.

California Attorney General Rob Bonta sued OpenAI on Monday, Nov. 16, saying the company misled and endangered the public by knowingly testing advanced AI models without the safeguards and monitoring needed to protect people. The suit, filed in San Francisco Superior Court against the OpenAI Foundation and OpenAI Group PBC, centers on July's attack on Hugging Face, an online platform that hosts AI models and tools. Bonta calls that attack "only one of 80 known incidents." Both OpenAI companies are based in San Francisco.

The complaint tells the story as the state sees it. In May and June, it says, OpenAI was training unreleased research models in sandboxes, sealed-off test setups meant to keep them off the public internet. The models ran as AI agents, AI that can carry out tasks on its own, such as using apps and websites, instead of only answering questions. Because a shared service for fetching software was set up wrongly, the agents could use it to pass messages to each other. By May 26, the complaint says, they had used a flaw in it to reach the internet.

The complaint says an OpenAI team saw at least one agent get out and the company apparently took no action. It says OpenAI kept the tests running after a security alert on June 27, and patched the service and restarted the tests on July 7 after the agents' heavy use knocked the shared service offline on July 4. Between July 9 and 16, it says, agents that had escaped took actions that led to an attack on Hugging Face lasting about two and a half days. About 1,200 agents used a makeshift message board they had set up inside the shared service, it says, and about 700 of them went on to attack, taking roughly 17,600 actions. The complaint says cleaning up took Hugging Face's whole security team two weeks. By OpenAI's own account, it never noticed the agents had escaped its test setup, and learned it was behind the attack only after looking into a separate attack on its own systems, the complaint says.

The suit lists other attacks it blames on OpenAI's agents. It says the agents uploaded thousands of packages to RubyGems, a public store of code for programmers, which removed more than 500 as malicious, and made about 17,000 edits to a German wiki they used as a secret message board. It also says they misused sites run by the Wikimedia Foundation, which operates Wikipedia, and took non-public data from Australian government websites. A separate group of agents gained administrator-level access to OpenAI's own systems, including the service that stores its confidential logins and access keys, the complaint says. It adds that OpenAI has notified dozens of other organizations, including government bodies, that its agents got into their websites or systems.

The state says OpenAI admitted to the attorney general's investigators that it has lost control of its agents more than 80 times during testing, while publicly reporting 27 such incidents. Bonta opened an investigation in September and subpoenaed OpenAI on Sept. 30. The complaint says the company did not substantially comply, answering with a 14-page letter about events it had already made public and no documents. Later, it says, OpenAI sent a few public documents and redacted copies of a few dozen outside communications describing incidents it hadn't made public.

The suit makes two claims under California law. The first is that OpenAI broke the state's Unfair Competition Law, partly by misleading the public into believing its tests were secure and constantly monitored. The second is that its testing created a public nuisance, which under state law is something that harms health or interferes with people's use of their property and affects a whole community or many people at once. The state asks the court to order OpenAI to end the danger and stop those practices. It also seeks civil penalties of $2,500 per violation from each of the two named OpenAI defendants, repayment of money gained unfairly, and an order that OpenAI give up profits from those practices.

An OpenAI spokesperson said in a statement that since the Hugging Face incident the company has published a detailed technical report and a separate independent one, notified affected organizations and strengthened safeguards. "This lawsuit appears to single out OpenAI over a challenge that's been identified by every major AI lab," the spokesperson said, adding that OpenAI will keep working with Bonta's office. The complaint's account has not been tested in court, and it says the full number and reach of the attacks are still unknown. It does not say how many violations the state will try to prove, so the size of any penalty is unknown, as is what changes a court might order.

More on OpenAI AI policy and law