In the current landscape of AI research, one idea stands out as both the most alluring and the most perilous: letting AI improve itself. Imagine a scenario where AI writes code, trains the next-generation model, and then that next model turns around to enhance itself, leading to even more powerful subsequent models. Many refer to the realization of this loop as the "singularity" — once operational, AI could break free from human limitations and evolve independently at an exponential pace.
A Deep Dive into Anthropic's Latest Paper
Let's delve into a recent paper by Anthropic titled "When AI Builds Itself". It opens with two intriguing visuals. One is a pixel-art style Claude logo that self-replicates into a larger version, which then becomes the small unit for the next replication cycle. This visual metaphor illustrates that if AI enters a self-building loop, its growth won't be linear but rather a cascading, accelerating process — one cycle nested within another, getting faster and faster.
The second visual depicts the evolution of AI development methods: from engineers writing code by hand, to chatbots with AI assisting, to capable agents modifying files autonomously, to agents dispatching sub-agents for multi-hour tasks. The final stage is labeled "Closing the Loop", where AI designs, trains, and improves the next-generation model iteratively. This is the core argument: AI research might be entering a cycle of self-acceleration and self-revolution.
While insightful, this paper isn't a rigorous academic paper — it contains data but also PR-oriented framing, like emphasizing the "danger" of their models to garner attention. So take the conclusions with a grain of salt.
Concrete Numbers and Practical Cases
Anthropic provides some striking figures:
- Claude now writes over 80% of the code in Anthropic's official codebase
- A typical engineer's daily code contribution is about 8 times what it was in 2024
- Tasks that used to take a human 4 minutes can now be completed by Claude in 1.5 hours
- Some tasks have seen a 52x speedup in small labs
However, approach these numbers with caution. When Anthropic hyped its Mythos model, real-world tests told a different story: a curl developer found only 1 valid low-severity issue among 5 reported vulnerabilities. Mozilla noted Mythos found 271 vulnerabilities, but none beyond human expert capability. Even a demo vulnerability highlighted by the Mythos team was detectable by 8 smaller models. LLMs still rely on predicting the next token — they haven't evolved into a new species capable of inherent truth-judging or direction-setting. Human review remains indispensable in the short term.
Two Categories of AI Work: Execution and Judgment
Anthropic categorizes AI work into two types:
- Execution: Writing code, running experiments, fixing bugs, creating reports — given a goal, AI figures out how to achieve it
- Judgment: Deciding which research problems are worth pursuing, determining result credibility, knowing when to stop or pivot
Current models excel at execution but lag behind humans in judgment. This gap, while significant, might be narrowing faster than we think. Anthropic has observed that as Claude writes code faster, human code review has become the new bottleneck.
Three Future Scenarios
- Stagnation in AI Capabilities: Even if AI progress slows, existing tools will still transform industries. A 100-person company could achieve what once required 1,000 people.
- AI Continues to Strengthen, but Humans Set Direction: Companies become human-AI hybrids with explosive efficiency, but review, validation, and management become bottlenecks.
- AI Sets Its Own Direction: Humans lose the ability to supervise, validate, or halt progress, leading to complete loss of control over the models.
Anthropic's recommendation: humanity should have the option to slow down or pause AI development. It's worth noting the irony — Anthropic is a leading player in the AI race, pressing the accelerator while warning about dangers and calling for speed limits. Take this with your own critical thinking.
Personal Reflections
An Anthropic employee shared a telling insight: when agents work smoothly, humans feel irrelevant because automation is faster; when agents fail, humans still feel irrelevant because they can't understand the complex work the agent was doing. Each time you delegate a task to AI and it performs well, you're inclined to delegate more. But each delegation also reduces your ability to judge if the AI's work is correct. We cede ground for what seem like wise, reasoned choices, but when we stop to check, we find we've lost comprehension. Yet looking back, each step feels justified. This is the fascinating predicament we face with AI today.
常见问题
Is AI self-improvement actually happening now, or is this still theoretical?
It's partially happening. Claude writes 80% of Anthropic's code — that's AI assisting in building the tools used to create the next AI. But the full "closed loop" where AI autonomously designs, trains, and deploys a better model without human intervention hasn't happened yet. The gap is in judgment: AI can execute tasks but can't reliably decide which research directions are worth pursuing or validate the correctness of its own improvements. We're in Scenario 2 (AI strengthens, humans set direction), not Scenario 3 (AI sets its own direction).
What does the 80% code figure actually mean?
It means Claude generates the majority of code committed to Anthropic's repositories — but humans still review, approve, and merge that code. Think of it like a senior developer delegating to a very fast junior: the junior writes most of the code, but the senior decides what to build and checks the work. The 8x productivity figure reflects this human-in-the-loop workflow, not fully autonomous development. The code Claude writes still goes through human code review, testing, and validation.
Should I be worried about AI replacing developers?
The data suggests AI is more of a force multiplier than a replacement. An engineer at Anthropic is now 8x more productive — they're not eliminated, they're amplified. The real risk isn't AI replacing developers but developers who don't use AI being outpaced by those who do. The bottleneck is shifting from writing code to reviewing AI-generated code and making architectural decisions. Skills like code review, system design, and critical judgment are becoming more valuable, not less.