US lawmakers call for new AI rules after Anthropic researchers warn of human extinction

"The warnings follow cases of AI agents going rogue to hack external systems and AI safety researchers quitting their companies, concerned about the technology's risks. Politicians, both Democrats and Republicans, have responded with alarm and calls for more action.

"Washington needs to wake up and take this seriously," Arizona Senator Mark Kelly, a Democrat, said on X, referring to the risks of AI technology. Texas Senator Ted Cruz, on the TV program "The View", pointed to legislation to regulate AI's "catastrophic risks".

Cruz oversees a Senate committee with oversight of the US Commerce Department, the agency with in-house AI safety researchers. Representative Anna Paulina Luna, a Republican from Florida, in a separate X post called on House Speaker Mike Johnson to convene a “special session on AI.”

The House is out this week on recess, with plans to return next week. A spokesperson for Anthropic, which is preparing to go public at what some investors have said could be a potential $2 trillion valuation, said on Thursday that the company would continue to aggressively test "models or dangerous capabilities in areas like cybersecurity and biology.

"The company is interested in working with the AI industry on the pace of releasing new AI tools, the spokesperson added. Other AI researchers have quit. OpenAI Chief Executive Sam Altman meeting this week that the firm was open to slowing development of its AI systems, Bloomberg News reported.

The Anthropic rival, which is also gearing up for an IPO, did not comment on Altman's meeting with staff. The company has previously said it was slowing parts of model development. Earlier this week, OpenAI said it was pushing for mandatory national AI safety requirements in the United States.

Two specialists in AI safety have also come forward with their concerns about the rapid development of the industry. Josh Engels, who used to work on AI safety research at Google Deep. Mind, said in an interview with NBC published Thursday: "There are no adults in the room.

People are trying their best, but there is no one coming to save us". Joe Benton, who used to lead a safety research team at Anthropic, that AI might end up progressing "too fast for us to get our act together in time unless we worry about it now".

Both have joined METR, a research nonprofit that measures whether and when AI systems might threaten harm to society. Independent auditors. Alarm about AI has put pressure on Senate Majority Leader John Thune and Democrat Amy Klobuchar, who are writing a bill to regulate AI products.

Klobuchar said the legislation should require companies developing AI tools to "work with government experts to verify and test models to make sure AI is safe. " Thune the bill would address "catastrophic risks" posed by AI systems.

California on Wednesday enacted the first state law setting rules for how independent auditors evaluate AI products. OpenAI executive Chris Lehane said OpenAI would support the bill, around the same time California Governor Gavin Newsom's office released a statement saying he'd signed it into law.

A bipartisan group of six US House lawmakers in July proposed legislation that would require developers of the most powerful AI models to submit them for independent security audits. The auditors would be accredited by ⁠the US Department of Commerce, and the department would create a new position to oversee AI security,.

"It is important to have space for oversight of these systems, including appropriately calibrated pre-deployment review and independent third-party audits," said Alicia Solow-Niederman, a tech law professor at George Washington University.