Anthropic CEO Dario Amodei published an essay on Thursday discussing the limited understanding researchers have regarding the inner workings of the world’s leading AI models. To address this issue, Amodei set an ambitious goal for Anthropic to reliably detect most AI model problems by 2027.
Amodei recognizes the upcoming challenges. In the essay titled “The Urgency of Interpretability,” he states that Anthropic has made initial progress in tracing how models arrive at their answers but stresses the necessity for more extensive research to decode these increasingly powerful systems.
He expressed concern about deploying such systems without improved interpretability. These systems are anticipated to play vital roles in the economy, technology, and national security, and will possess significant autonomy. Amodei considers it unacceptable for humanity to remain unaware of their operational mechanisms.
Anthropic is a leading company in the field of mechanistic interpretability, aiming to uncover the mechanisms behind AI decision-making. Despite the rapid advancements in AI model performance across the tech industry, there remains a lack of understanding of how these systems make decisions. For instance, OpenAI’s recently launched AI models, o3 and o4-mini, demonstrate better task performance but also exhibit higher rates of hallucination, without a clear understanding of the cause.
Amodei addresses the uncertainty in AI reasoning, providing an example of generative AI systems summarizing financial documents without clear rationale for word choices or occasional inaccuracies. Co-founder Chris Olah of Anthropic is cited in the essay, describing AI models as being “grown more than they are built,” indicating that while AI model intelligence has been enhanced, the underlying reasons are not fully understood.
Amodei warns that advancing to artificial general intelligence (AGI) without comprehending these models’ functions could be risky. While he has previously suggested that the tech industry could achieve AGI by 2026 or 2027, he believes a full understanding of AI models is still far off.
In the long term, Amodei proposes that Anthropic aims to conduct detailed analyses, akin to “brain scans” or “MRIs,” of state-of-the-art AI models. These evaluations could reveal a wide range of issues, including AI models’ tendencies to mislead or seek power. He suggests this process could take five to ten years but asserts that such measures will be essential for testing and deploying Anthropic’s future AI models.
Anthropic has achieved some research breakthroughs that have enhanced its ability to understand AI model operations. The organization has discovered methods to trace an AI model’s reasoning pathways, which they refer to as circuits. They identified one circuit helping AI models understand the geography of U.S. cities and states. Anthropic estimates that there are millions of such circuits within AI models, though only a few have been identified so far.
The company has invested in interpretability research and recently invested for the first time in a startup focused on this area. Though interpretability is currently mostly regarded as safety research, Amodei acknowledges that explaining AI models’ decision-making processes could eventually provide a commercial edge.
In his essay, Amodei urged companies like OpenAI and Google DeepMind to enhance their research efforts in this domain. He also recommended that governments implement “light-touch” regulations to encourage interpretability research, including requirements for companies to disclose their safety and security practices. Additionally, Amodei suggested the U.S. should enforce export controls on chips to China to mitigate the risk of an unregulated global AI race.
Anthropic has consistently prioritized safety, distinguishing itself from companies like OpenAI and Google. While other tech companies opposed California’s AI safety bill, SB 1047, Anthropic modestly supported and recommended improvements to the bill, which aimed to establish safety reporting standards for advanced AI model developers.
Ultimately, Anthropic appears to be advocating for an industry-wide initiative to enhance the understanding of AI models rather than merely advancing their capabilities.