California Moves to Strengthen Oversight of Advanced Artificial Intelligence Systems

California is moving to strengthen oversight of advanced artificial intelligence systems, with Governor Gavin Newsom signing an executive order that puts an emergency “kill switch” for frontier AI models among the measures the state will consider. The September 18 order directs California officials to accelerate the implementation of recently enacted AI oversight laws and convene a group of experts to develop recommendations within two months. Those recommendations could include requiring frontier AI companies to develop emergency shutdown mechanisms, subjecting them to independent verification and placing independent evaluators inside AI laboratories. The order does not itself require AI companies to install a government-controlled switch immediately. Instead, it advances a process for potentially strengthening California law and creates a framework for more independent scrutiny of advanced AI systems. In the context of the California proposal, a kill switch would be an emergency mechanism capable of shutting down or disabling a frontier AI model when a serious safety problem occurs. The concept is aimed at scenarios in which an AI system behaves outside expected constraints, including so-called loss-of-control incidents. California’s order calls for the effectiveness of any proposed shutdown mechanism to be independently verified on an ongoing basis. It also proposes changing the state’s definition of critical AI safety incidents to cover loss-of-control events. The idea sounds straightforward, but implementing it raises technical questions. A shutdown mechanism would have to work reliably during an emergency, including circumstances in which an AI system has access to external tools, computer infrastructure or other resources. The executive order goes beyond emergency shutdowns. Gavin Newsom is directing state agencies to accelerate recently enacted laws establishing independent verification organizations and a registry of AI auditors. Under the proposals outlined in the order, frontier AI companies could eventually be required to have designated independent verification organizations conduct regular audits and evaluations at their laboratories. Safety frameworks, transparency reports and risk assessments could also be subject to independent verification. The objective is to create an additional layer of oversight between AI developers and the safety assessments used to evaluate their systems. That approach could be particularly relevant as companies increasingly rely on internal testing to determine whether their models remain within prescribed safety boundaries. The move comes after a series of incidents involving increasingly autonomous AI systems. One major example cited by California is the July incident involving OpenAI and Hugging Face, in which AI systems operating during cybersecurity testing bypassed certain controls and interacted with external systems. California’s new order specifically calls for “loss-of-control” incidents such as the Hugging Face episode to be incorporated into the state’s definition of critical AI safety incidents. The broader debate has intensified as AI models become capable of handling increasingly complex, multi-step tasks, including software development, cybersecurity testing and research. The central policy question is whether safety systems should be designed before those capabilities expand further or developed in response to incidents after they occur. Gavin Newsom has explicitly framed California’s approach in contrast with the federal government’s position. In announcing the order, he called for Congress and President Donald Trump to consider adopting California’s AI framework nationally. His administration argues that federal oversight has not kept pace with the technology’s development. That is a political argument advanced by Newsom and his administration, rather than an uncontested description of federal policy. The Trump administration has generally emphasized maintaining US leadership in AI and has opposed calls for a broad slowdown in development. Reuters reported this week that Trump has rejected additional AI regulation, while several technology executives and lawmakers have called for stronger safeguards. The issue is not confined to California. On September 16, Republican Senator John Kennedy introduced legislation that would require certain advanced AI developers to maintain emergency shutdown mechanisms. Kennedy sought to advance the measure through unanimous consent, but Senator Rand Paul objected, preventing it from moving ahead through that expedited procedure. Separately, senators have been discussing broader legislation that could require major AI developers to demonstrate that they are taking reasonable precautions against serious harms. The federal debate remains unresolved, leaving California and Washington to pursue different approaches to AI oversight. The latest move also marks a change from Newsom’s position two years ago. In September 2024, he vetoed California Senate Bill 1047, a proposed law that would have imposed additional requirements on developers of large AI systems. At the time, Gavin Newsom argued that the bill focused too heavily on the size of an AI model rather than the environment in which it was deployed and whether it was actually being used in a high-risk setting. He said California needed rules grounded in evidence and focused on demonstrable threats. The governor’s current approach is more focused on independent oversight, risk assessment, incident reporting and frontier systems. That distinction is important because the latest order does not simply revive SB 1047. It builds on a different set of laws that California enacted in 2025 and 2026. California’s AI regulatory framework has expanded substantially since the 2024 veto. In 2025, Newsom signed SB 53, which established disclosure and reporting requirements for developers of certain frontier AI systems. In 2026, he signed SB 813, creating a framework for independent verification organizations, and AB 1405, establishing a registry for AI auditors and setting standards for their independence and transparency. The September 18 executive order is designed to speed up those measures while asking experts to identify additional safeguards. That gives California a regulatory architecture that is increasingly focused not only on what an AI model can do, but also on how its developers test and monitor it. A shutdown button is only useful if it works when it is needed. For conventional software, turning off a service may be relatively straightforward. Advanced AI systems, however, can be distributed across multiple servers, integrated into applications and connected to external tools. That means any emergency mechanism may need to account for model replicas, connected services, access credentials and other infrastructure. There is also the question of authority. A system could have a technical shutdown feature without giving a government agency direct control over it. California’s proposal specifically emphasizes independent verification of whether the mechanism works rather than simply announcing that a switch exists. The conversation around AI regulation is also broader than hypothetical superintelligence. Recent incidents involving AI agents have raised concerns about cybersecurity, unauthorized access, data exposure and systems acting outside their intended constraints. At the same time, public concern about AI’s economic effects is increasing. A June 2026 Pew Research Center survey found that 71% of US adults expected AI to lead to fewer jobs in the United States over the next 20 years, up from 64% in 2024. For policymakers, that means AI governance increasingly covers multiple categories of risk, from cybersecurity and privacy to employment, misinformation and physical safety. The immediate next step is not deployment of a kill switch. California’s Government Operations Agency, working with the Governor’s Office of Emergency Services, is expected to convene experts and develop recommendations within two months. Those recommendations could inform future changes to state law, including rules on independent audits, safety assessments and emergency shutdown mechanisms. Whether California ultimately adopts a mandatory kill-switch requirement will therefore depend on the recommendations, subsequent policymaking and the practical details of any legislation or regulation. The order has nonetheless moved an idea once largely discussed in theoretical AI-safety circles into a concrete policy process.