Commentary: Anthropic’s call for AI development pause deserves attention. It also raises questions

SINGAPORE: When one of the world’s leading AI firms sounds an alarm on the very technology it is building, it is only natural to pay attention.

Earlier this month, Anthropic called for a worldwide pause in frontier AI development. It warned of a phenomenon called recursive self-improvement - the threshold at which an AI system can autonomously design and build more capable versions of itself without human help.

In a lengthy report on Jun 4 titled When AI Builds Itself, Anthropic is careful to say this threshold has not been crossed and is “not inevitable” but warns it could arrive sooner than most are prepared for.

This is not to say that we will face a science-fiction scenario of a suddenly malevolent machine. But the risk is when the pace of AI capability gain becomes faster than what humans can track, there may be no opportunity to intervene - not because AI chooses to resist correction, but because the pace has simply outrun the human capacity to understand, evaluate or redirect what is being built.

This is why Anthropic’s call deserves to be taken seriously but taking it seriously also means asking a question that the firm raises and leaves unanswered - a pause toward what, exactly?

A CALL FOR PAUSE WITH NO EXIT CRITERIA

The suggestion of a pause isn’t new. In 2023, non-profit Future of Life Institute called for a six-month halt in AI development to allow time for safety guardrails.

Anthropic’s reiteration for a slowdown to allow societal structures and alignment research to keep up is a reasonable aspiration, but it has done so without a plan.

What does alignment research need to achieve before frontier AI development can safely resume? What level of interpretability - in this case, our ability to understand what is happening inside these systems - would be sufficient? What governance architecture would need to exist, verified by whom, before the pause can end?

Anthropic’s report acknowledges that a credible pause must specify what triggers it, what lifts it and who adjudicates but proceeds without answering any of those questions. The pause button is described but the exit criteria are absent. Without which, a pause is merely a deferral. Alignment research remains unsolved; interpretability tools remain immature; international oversight institutions don’t yet exist. If none of those conditions are specified as requirements for resumption, what exactly is the pause buying?

AN ENFORCEMENT PROBLEM

The enforcement problem compounds this. Anthropic draws the analogy to nuclear arms control treaties but acknowledges it breaks down quickly since AI training is “far easier to conceal than missile silos”.

The “incentive to defect quietly is enormous” as well “because whoever continues while others pause could inherit the lead”, its report said.

What the report doesn’t fully reckon with is that the nuclear analogy cuts against the proposal in a deeper way. Arms control succeeded not by pausing physics research but by managing deployment through verified treaties, mutual inspection regimes and a balance of power that gave all parties a structural interest in compliance. Those regimes took decades to construct and still have significant gaps.

Verification of AI treaties faces challenges with significant similarities to nuclear arms control - restricting both government and corporate activity, protecting sensitive information, countering well-resourced adversaries - while states are currently far less willing to accept the costs that meaningful verification would require.

The geopolitical context makes this structural problem acute. AI capability is now central to US-China strategic competition in ways that make coordinated compliance not merely politically difficult but strategically irrational for either party.

In a recent example, the US government ordered Anthropic to suspend access to its most advanced AI models for foreign nationals, citing national security concerns.

All these suggest that a pause is structurally unlikely.

FURTHER SCRUTINITY NEEDED

It is also worth noting that Anthropic’s pause proposal arrived with two details that warrant further scrutiny.

The company had confidentially filed for an initial public offering just days earlier, raising the question of why a lab heading for a potential mammoth listing would simultaneously urge hitting the brakes on the industry driving its valuation.

On the same day the pause proposal was published, a Financial Times report said Anthropic had embedded engineers with the US National Security Agency to deploy its powerful Mythos AI model for offensive cyber operations. This may make its call for cooperative global AI governance somewhat harder to read as straightforwardly altruistic.

Neither detail is necessarily disqualifying but together they illustrate the central problem - the actors best positioned to advocate for a pause are simultaneously those with the greatest financial and strategic stakes in continued development. Any governance framework built primarily around the preferences of incumbent frontier labs should be scrutinised with that in mind.

THE OTHER ASPECT THAT NEEDS ATTENTION

The latest development has raised an important question about the future of AI development, with the standard answer being better technical guardrails, stronger international frameworks and improved verification mechanisms.

These are necessary but there is another equally important question that remains unasked: what AI adoption is already doing to the humans who must govern it.

Every governance regime, however well-designed, ultimately depends on human and institutional capacity to implement, monitor, and enforce it. That capacity is being actively shaped right now - not by frontier models approaching the risk of recursive self-improvement, but by the current generation of AI systems already deeply embedded in how policymakers research, how analysts evaluate evidence, how regulators reason through complex technical questions, and how the next generation of experts is trained.

This is not an argument against AI adoption. It is an argument that adoption, like development, requires governance too. Yet, the conditions under which adoption builds, rather than gradually substitutes, human capability are not currently the subject of serious policy attention.

If mass AI adoption is quietly restructuring the cognitive habits and judgment of the people who would eventually staff and operate AI governance institutions, then the governance infrastructure Anthropic wants to build has a problem that no pause addresses. You cannot pause your way to better human judgment.

This matters for Singapore and the ASEAN region. We are not frontier AI developers and the pause debate is not ours to drive. But how this region structures AI adoption - in education, in public administration, in professional practice - will determine whether we retain the institutional and human capability to participate meaningfully in AI governance at all, or whether we arrive at that conversation having already quietly outsourced the judgment it requires.

Jungpil Hahn is a Professor in the Department of Information Systems and Analytics and a Provost's Chair Professor at the School of Computing at the National University of Singapore.