The cat is out of the bag.Simmering concerns about AI agents going rogue, plotting against their owners and hacking outside companies, and new systems coming perilously close to being uncontrollable by humans, are boiling over.
Industry insiders warning that AI bots might kill us all are stoking our worst nightmares.President Trump needs to take these fears about AI seriously, work with tech leaders to establish credible and enforceable safety measures, and get ahead of Democrats (and some Republicans) in Congress who want to regulate the industry.For the economy and our national security, nothing could be more important.Evan Hubinger, who describes himself as a "science alignment lead" at Anthropic, posted on X last week that he thinks the odds are better than 10% that AI could kill all humans within the next decade.
He was responding to a post from colleague Jacob Coxon, who just quit his job at Anthropic and posted by way of explanation: "The people building AI earnestly believe that it could kill us all by the end of the decade.This is not a marketing stunt.
If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but, I hear the same people express fear after a colleague quit over safety fears."RESEARCHER WHO DEPARTED AI ROLE OVER FEARS OF RACING INTO EXTINCTION SAYS PEOPLE ARE 'BEGGING' FOR REGULATIONHubinger wrote soon after: "To be clear, as we say in our latest Risk Report, I think the risk from present models is low.What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought."These posts gathered tens of millions of views and attracted concurring comments from other industry insiders.
"Recursive self-improvement" means AI agents creating newer, better agents; once we cross that threshold, and we are apparently very, very close, it is easy to imagine those same agents surpassing human capabilities and allowing, in effect,...