'They want to build a moat': Anthropic's scary warnings about rapid AI 'self-improvement' and 'temporarily' pausing…

Anthropic suggests a "slowdown or pause" on AI

AI could soon start developing itself, the company warns

There have been mostly negative responses online

Even the giant companies at the forefront of AI are suggesting it might be time to take a step back and think about where the tech is heading: Claude developer Anthropic has posted a lengthy new blog post in which it suggests "a meaningful slowdown or pause" while we all take a breath.

Authored by Anthropic executives Marina Favaro and Jack Clark, the post centers on the idea of AI developing itself, known as 'recursive self-improvement' — at which point it might get out of our hands very quickly, as AI takes over the business of designing and developing its own models and interfaces.

"We are not there yet, and recursive self-improvement is not inevitable," explains the post. "But it could come sooner than most institutions are prepared for." By next year, an AI model like Claude could be capable of completing tasks that would take a human coder weeks, according to Anthropic's calculations.

As per Anthropic's internal testing, Claude is managing more code than ever before, and is rapidly getting better at writing code that both works and can be understood by human engineers. In fact, Claude is now catching bugs that the best programmers at Anthropic previously missed.

While humans still do better at seeing the big picture and context outside the current task, the blog post says, this is something else that AI may soon catch up with — though this part is less clear. If and when that does happen, AI could escape our control, and that's where the proposals about a freeze on AI development come in.

A growing anti-AI sentiment

Anthropic calls for global freeze in AI development from r/technology

"If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing," writes the Anthropic team. "But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe."

With that in mind, the blog post floats the idea of a "global coordination mechanism" to slow down or pause frontier AI development — which would of course require competitors like OpenAI and Google to promise not to try and jump ahead in secret. During the pause, more of the future potential of AI and possible safeguards could be worked out.

Sign up for breaking news, reviews, opinion, top tech deals, and more.

Anthropic says it will "organize conversations" with governments, researchers, and AI companies to see if it's going to be possible to put the brakes on development at this point. If we don't do it now, we might lose the opportunity altogether, which is why Anthropic wants to ask these questions now.

Out in the flesh and blood world, a noticeable anti-AI sentiment is growing — at least outside of Silicon Valley and coders. Online reactions to Anthropic's post have accused the company of wanting to protect its own lead in the market and raise hype for its upcoming IPO — though many share the same safety concerns raised in the blog post.

"It's regulatory capture," says one commenter, pointing out that this is something AI companies regularly do. "They want to build a moat around their business." Another Reddit post frames the scenario in a more vivid way: "Wolf with a bloody maw and engorged belly says it's time to stop eating meat for a little bit."

Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.

Dave is a freelance tech journalist who has been writing about gadgets, apps and the web for more than two decades. Based out of Stockport, England, on TechRadar you'll find him covering news, features and reviews, particularly for phones, tablets and wearables. Working to ensure our breaking news coverage is the best in the business over weekends, David also has bylines at Gizmodo, T3, PopSci and a few other places besides, as well as being many years editing the likes of PC Explorer and The Hardware Handbook.