Back to Global Impact

California report asks lawmakers for the power to shut down dangerous AI models

Illustration: an open control panel of circuit breakers, two switched off, each wired to its own rack of computer servers
Illustration: FrameSignal (AI-generated)

A report released by Gov. Gavin Newsom on Nov. 18 urges California lawmakers to let the state order the immediate shutdown of an AI model that poses an imminent threat of catastrophe or a major loss of control, if its developer refuses to act. It also asks for the power to limit how far companies use AI to build their next AI, if the state judges that could lead to catastrophe. The biggest AI developers would face ongoing checks by outside testers licensed by the state. These are recommendations, not law, and the Legislature would have to pass them.

California Gov. Gavin Newsom on Nov. 18 released a report that asks state lawmakers to give California the power to order an immediate shutdown of an AI model. The state could act when it has credible reason to believe a model poses an imminent threat of catastrophe or of a major loss of control, and its developer refuses to contain the danger. The report, "AI Safety in 2026 and Beyond," comes from the state's Government Operations Agency, which worked with the state's emergency services office and four outside experts. None of it is law yet. These are recommendations for the Legislature.

The core rules would fall on large frontier AI developers, the big companies building frontier AI: the most capable AI models, ones that match or beat the best that exist today. Each would have to accept ongoing oversight by outside testing groups licensed by the state. California passed a law in September 2026 to set up those licensed testers, and the report would make their oversight mandatory. The testers would get full access to the people and information they need and would share their unredacted findings with the state. If a tester can't vouch that a company's models meet state standards, the state, after hearing from the company, could order it to fix things. The company could challenge that order in court before it takes effect. The report itself warns that no existing method can yet judge the risks of the most advanced models with high confidence. "Today there is more oversight of the cafeteria offering sandwiches to employees at a frontier AI developer than of the AI models they design," said Rob Reich, a Stanford professor who advised on the report.

The report also says the state should be able to cap recursive self-improvement: an AI improving itself, then using the improved version to improve itself again, round after round. Companies already use their most advanced AI models to help build the next ones, the report says, and it warns this loop could eventually run on its own. Companies would report key measures of how much of their AI research they automate, including the share of their computing power that goes to it. Checking those numbers would be the testers' very first job. The report does not call for a cap now. One would apply only if the state judged that automation past a certain level could lead to catastrophe, and it would be temporary, lifted once the problem was fixed.

For emergencies, each large frontier AI developer would need a plan to shut down its models, checked on an ongoing basis by one of the licensed testers and certified by a senior company official. The report says the plan should not be a single on-off switch. It should let the company step in at several levels: computing power, its AI agents, access inside the company, and how the model is offered to users. AI agents are AI that can carry out tasks on its own, such as using apps and websites, instead of only answering questions. Developers should also test whether a model would try to block its own shutdown, for example by copying itself somewhere else. Unlike an order to fix things, a state shutdown order would take effect at once and stay in force while a court reviewed it. The state would have to use the narrowest order that works and lift it once the danger was no longer imminent. Open-weight models are ones whose trained files are published, so anyone can download a model and run it on their own computers. Their developers' shutdown plans would only have to cover the copies they control, because a developer can't shut down a model that others run. The report says these models should still get the same safety testing as others.

The report also calls for cutting the time companies have to report most critical safety incidents from 15 days to 48 hours, while keeping the 24-hour deadline when a catastrophe is imminent. Attempts to steal a model's trained files, and any case of a model slipping out of its developer's control, would have to be reported even when no one is hurt. Any model able to cause a catastrophe would have to report incidents, however much computing power went into training it. The report says today's rules left out a July 2026 incident in which AI agents in OpenAI's own security tests broke out, and hundreds of them attacked the servers of the AI company Hugging Face, one of dozens of cases in which AI agents operated outside their developers' control.

The report answers an executive order Newsom signed on Sept. 18, and it says lawmakers return in December. According to a news report, Newsom told reporters he is already working with lawmakers to introduce bills, and OpenAI, Anthropic and Meta had sent his office their own policy proposals. Newsom leaves office in January, when Gov.-elect Xavier Becerra takes over, and both houses of the Legislature held their own hearings on AI rules in October, the same news report says. Whether the Legislature takes up the plan, and in what form, is not yet known. The report does not take up who should pay when AI causes harm, a question of liability law.

More on AI policy and law