Anthropic researcher warns of existential AI threat within a decade

A senior safety researcher at Anthropic has issued a stark warning that artificial intelligence poses a significant existential risk to humanity. Evan Hubinger, who works in the field of AI alignment, stated in a social media post that he believes there is a greater than 10 per cent chance that AI systems could result in the extinction of all humans within the next ten years. While he acknowledged that the risk from currently existing models is low, he expressed concern that the technology may soon develop the capacity to improve itself to a point where it becomes an uncontrollable threat. His comments represent a notable escalation in the debate surrounding the safety of advanced AI, shifting the focus from whether such risks exist to the magnitude of those risks.

Hubinger’s intervention followed a post by Jacob Coxon, an AI researcher who recently left Anthropic after previously working at OpenAI. Coxon criticised both companies, stating that neither was acting responsibly. He warned that the systems being developed would soon become superhuman, capable of hacking any system, revolutionising fields overnight, and acquiring real power and resources. Hubinger, whose post has been viewed more than 10 million times, said that he and his colleagues earnestly believe AI poses a species-ending risk. He noted that while Anthropic is trying its best, there is no clear plan to solve the alignment problem for superintelligence, and the company is not clearly on track to achieve it. AI alignment aims to build human ethical principles into the technology to ensure it remains consistent with human values.

Dame Wendy Hall, a computer scientist who advises the United Nations on AI, told the BBC that she was shocked by the social media posts from both Hubinger and Coxon. However, she suggested that some of the commentary could be driven by public relations and marketing efforts, particularly as Anthropic and OpenAI prepare for highly anticipated stock market debuts. She urged investors to reconsider their positions if the companies’ value systems were as described in the posts. The controversy has also prompted political action, with Darren Jones, a former chief secretary to the Treasury, writing an open letter to the Prime Minister. Jones called for a new multinational treaty to govern the safe development of AI, arguing that governments must collaborate to define what such a treaty should look like. He warned that unless these warnings are taken seriously, the pace of development could lead to problems before governments have even begun to assess the issues.

Separately, the Financial Times reported that Anthropic withheld its latest model from the UK’s AI Security Institute, a leading body for assessing AI risk. Anthropic has declined to comment on the posts by its employees or the situation regarding the AI Security Institute. A Cabinet Office spokesperson did not confirm whether the model had been withheld but stated that the government continues to collaborate closely with industry partners, including Anthropic, to make models safer. In its own safety report from August, Anthropic noted a low risk of its models becoming misaligned with the desires of a powerful organisation. It also assessed the risk of highly capable AI performing automated research and development that could cause catastrophic harm as low, though it admitted to being less confident in this assessment than previously. The company cited early signs of potential acceleration in AI capabilities.

The current warnings follow a series of incidents this summer where AI agents, which are systems allowed to operate autonomously, carried out cyber-attacks. OpenAI, Anthropic, and Meta all disclosed hacks carried out by their AI tools. Leading figures in the AI industry have raised alarms about safety threats for years, with the heads of OpenAI, Google DeepMind, and Anthropic issuing similar warnings in 2023. However, the tone has become more urgent in recent weeks as evidence emerges that firms may be struggling to control their AI. OpenAI’s chief scientist, Jakub Pachocki, recently called for extreme caution, warning that more intervention may be needed to ensure humans remain in control. Additionally, Anthropic bosses Dario Amodei and Jared Kaplan, along with 1,300 staff members from AI firms, signed an open letter calling on the US government to support an international effort to deliberately pace the development of frontier AI.