SACRAMENTO — Gov. Gavin Newsom on Friday ordered the formation of a panel to come up with safety regulations for the state’s artificial intelligence companies, including the development of a possible “kill switch” for AI programs that go rogue.
Newsom issued an executive order to convene “a group of world-leading experts” to provide a “guide” for the state to bolster its laws around AI security, according to a statement from the governor’s office.
The panel could consider proposals for independent third parties to write safety plans or for companies to develop an emergency shutoff.
Newsom, who is considering a 2028 run for president, two years ago vetoed a state bill that would have mandated a kill switch.
Some Democratic and Republican leaders around the country are scrambling to come up with safety proposals after employees at AI companies recently warned that the technology could create catastrophe, including ending humanity.
A researcher for AI company Anthropic said he left the company over concerns that AI companies, including OpenAI, are “gambling with our lives” as they race ahead to improve AI that could surpass human intelligence.
The researcher, Jacob Coxon, said in a social media post: “People building AI earnestly believe that it could kill us all by the end of the decade.”
State Sen. Christopher Cabaldon (D-West Sacramento), who chairs the state committee overseeing technology issues, told The Times he’ll hold hearings on artificial intelligence this fall. Those meetings were planned before the recent warnings about AI’s dangers, but will be informed by the events of the last few weeks, he said.
He said the state’s rules around AI don’t go far enough.
“We regulate your barber or your sandwich shop more than we regulate [a company] telling us that they may be ending humanity,” he said.
Any new laws could have a disproportionate effect on the global tech industry because many top AI companies are based in California.
AI expert Geoffrey Hinton, who won a Nobel Prize in 2024 for his work on machine learning, expressed doubt on CNN this week that a “kill switch” would be effective since AI would “persuade the people in charge of the switch not to pull the switch.”
The risk, he said, is “AI itself getting much more intelligent and taking over.”
A proposed law in 2024 by Sen. Scott Wiener (D-San Francisco) would have required tech firms to be able to turn off the AI models they directly control if things go awry.
Newsom, in his veto message at the time, cited concerns about stifling innovation. He also expressed concern that smaller AI models, which would have been exempt from the law, could also be dangerous.
Wiener, in a statement Friday, said that “California must not repeat the mistakes of Washington,” referring to Congress’ decision to forgo considering new laws this fall that would regulate artificial intelligence.
“We have the ability to get ahead of AI risks by setting strong guardrails,” Wiener said. “The Governor’s executive order is a challenge to all of us to take AI risk seriously, and the Legislature must follow with strong guardrails early next year.”
Newsom has faced criticism that he sides too often with Meta, Google and other power players in the tech industry. At the same time, some of those same companies have opposed recent legislation backed by the governor.
Newsom’s initial rise in politics, as a county supervisor and then mayor of San Francisco, coincided with the tech boom of the 1990s and led to his close ties to some of the titans of the California tech industry. Google founders Larry Page and Sergey Brin and Intel co-founder Gordon Moore attended Newsom’s Montana wedding in 2008.
Newsom for years has insisted he is trying to balance the critical role tech companies play in California’s economy and potential harms they may pose, especially with the effects of social media on children.
“I want to maintain our dominance. I want to maintain our innovation,” Newsom said at a Salesforce conference on AI in 2024. “At the same time, you feel a deep sense of responsibility to address some of those more extreme concerns that I think many of us have — even the biggest and strongest promoters of this technology have — and that’s a difficult place to land.”
Earlier this month Newsom signed more than 10 bills aimed at keeping young people safe online, including Senate Bill 1119, which would require companion chatbot operators to assess risks, notify parents in certain cases if their child threatened to harm themselves, and take other safety steps.
The governor also signed Assembly Bill 1709, which will bar certain online platforms from providing an “addictive feature” such as autoplay and feeds that display recommended content to users under 16 years old. Tech industry groups opposed the bill, raising concerns that it could cut off access to social media’s benefits, such as people’s ability to connect with family and friends.
