The previous chapters built the Agent’s capability system from different dimensions. Chapter 2 on context engineering laid the foundation for information management (including on-demand loading via the Skills mechanism); Chapter 3 on knowledge bases and user memory achieved cross-session knowledge persistence; Chapter 5 demonstrated how a Coding Agent can accumulate experience through the file system; and Chapter 7 on reinforcement learning post-training solidified strategies into model parameters. Each of these techniques has its own emphasis, but they all converge on one question: how does an Agent keep getting stronger?
Even the most advanced model, faced with one company’s refund process, one carrier’s sales script, or the calling convention of an obscure API, is as lost as a new hire on day one. Changing model weights takes massive data and compute, with update cycles measured in weeks—while in the real world, new APIs launch, old services shut down, and user needs shift constantly. An Agent needs a lighter, more immediate way to evolve: one that keeps expanding its capability boundary without touching the model parameters.
This chapter explores exactly that mechanism: Agent Self-Evolution. Self-evolution is externalized learning, encompassing two dimensions—distilling knowledge from experience, and proactively discovering and creating new tools. The core idea is to separate knowledge and processes from model parameters and transient context, externalizing them into persistent, retrievable, and reusable external resources—tool libraries and knowledge bases. This is not a replacement for post-training, but a complement: post-training addresses “how to make the model smarter,” while self-evolution addresses “how to make the Agent more capable.”
That is the practical case. But there is a more fundamental question: if the context window were infinitely long, and we stuffed every conversation and tool result an Agent has ever seen into it, would it automatically learn everything?
The answer is no, and the reason lies in the attention mechanism from Chapter 2. It is this chapter’s theoretical starting point, and after several chapters away, worth a brief review.
Chapter 2 stressed this repeatedly: the internal mechanism of in-context learning is closer to retrieval than to reasoning. Attention excels at lookup—“What cat is in the 37th cage?” is a one-step hit—but it cannot do inductive statistics in a single forward pass: “How many black cats are there across 100 cages?” requires traversing every record while maintaining a running count, which is thinking, not retrieval. Dump raw experience into the context and the model can “remember” it, but it will not “distill” it into reusable patterns on its own. Even a truly infinite context would not close this gap: the information would be there, but no one would perform the compression from “specific records” to “general patterns” on the model’s behalf. Worse, as Chapter 2’s “context rot” showed, the longer and noisier the context, the more attention gets diluted and the harder key information is to retrieve—an infinite context brings no automatic learning; it just lets retrieval quality decay. Karpathy’s insight is best read in reverse: the model’s “poor memory” is a feature, not a bug. It forces us to distill knowledge actively and explicitly, rather than hoping the model will divine patterns from its own sprawling history. In one sentence: learning does not happen automatically; it must be explicitly designed—and that is why this chapter exists.
Explicitly designed learning does not make its first appearance in Chapter 8. Earlier chapters planted several seedlings, though most serve immediate needs within a single session or across adjacent sessions. Chapter 2’s context compression spends an extra LLM call to swap bloated raw records for computed conclusions, supplying the “distillation” half that attention lacks; Chapter 2’s Agent status bar, where code deterministically maintains key conclusions in the context, is the other side of the same coin. Chapter 3’s user memory already pushes learning across sessions: the Agent builds up its understanding of the user over many conversations, and offline organization makes that understanding steadily more accurate.
User memory is itself a form of learning, but what it distills is information—who the user is: preferences, facts, habits. Chapter 8 fills in the other, longer-term half: distilling the problem-solving strategies, operating procedures, failure lessons, and even brand-new tools discovered through exploration into persistent, retrievable, reusable capabilities—so that the Agent doesn’t merely remember more, but can do more. This kind of learning plays out over a longer horizon and must be proactively initiated by the Agent, which is why it earns a chapter of its own. We start by placing it in the larger picture.
Three Learning Paradigms and the Positioning of Self-Evolution#
The three paradigms introduced in Chapter 1 (Figure 1-1) are used here only for positioning comparison. Post-training modifies model weights, solidifying “experience” into “muscle memory” through RL, offering high success rates and low latency, but with high update costs and long cycles (detailed in Chapter 7); In-Context Learning (ICL) provides demonstration examples in the prompt for temporary adaptation, with low cost and quick results, but it disappears when the session ends (see Chapters 1 and 2); Externalized Learning is the path most easily overlooked by developers—distilling knowledge into files, knowledge bases, and tools outside the model, which is persistent, interpretable, and modifiable at any time. The three are synergistic, not competitive: factual knowledge goes to RAG (see Chapter 3) and externalized storage, stable behaviors and formats are solidified by post-training, and current transient information is handled by in-context learning.
This chapter focuses on the path that does not change model weights—externalized learning, which corresponds to the two dimensions mentioned at the beginning of the chapter: externalizing experience into knowledge and Skills, and externalizing capabilities into tools. (This should be distinguished from Chapter 5’s “Code Creating Code: Agent Bootstrapping,” which is about Agents creating systems similar to themselves; this chapter is about capability growth without changing weights. Chapter 3 solves “how to store and retrieve” knowledge bases; this chapter solves “who fills and updates them”—how the Agent proactively accumulates experience.)
Why is it needed? Start with a cautionary scenario. A customer service Agent handles a certain bank’s refund process for the first time: after 15 minutes of exploration—3 phone calls, 2 different scripts—it finally succeeds. Without externalized learning, the next identical request costs another 15 minutes of the same exploration; everything learned this session dies with the session. The key word is “autonomous”: no human engineer writes documentation for the Agent—the Agent itself summarizes experience, builds tools, and updates the knowledge base in the course of doing its job, the way a veteran customer service rep turns scattered refund rules into a handbook she consults anytime and revises herself as new cases come in. The core philosophy: rather than expect the model to remember everything, spend extra compute after each task to summarize, compress, and structure the experience, then store it in a persistent, retrievable external system. Against parameter learning, this distills interpretable, verifiable, correctable knowledge quickly and without expensive training; against in-context learning, active distillation and structured organization spare the Agent from digging through mountains of raw records—and the result persists across sessions.
Figure 8-1: Externalized Learning Cycle
More importantly, externalized learning lifts the Agent from “remembering information” to “building capabilities.” It can summarize experience into general knowledge and store it for later retrieval (the RAPTOR tree-based summarization from Chapter 3’s RAG section works just as well for distilling experience layer by layer—from specific operation records to rules, and from rules to principles), and it can also encapsulate repetitive procedures into precisely executable tools, forming an ever-growing skill library. Consider a customer service Agent helping a client with a refund: it might learn three things of quite different natures. The first is a specific rule—“Company A’s refunds require verifying the last four digits of the credit card”—factual knowledge that belongs in the knowledge base. The second is a general-purpose tool—“use API X to query order status automatically”—a stable, reusable operation sequence, most economically frozen into a code tool. The third is a job manual—the complete Skill for the refund process—full of strategic judgment and ever-shifting business rules, better captured as a Skill document. Table 8-1 summarizes these three products of externalized learning.
Table 8-1 Three Products of Externalized Learning
Product Form
Content Carried
Example
Usage Method
Knowledge Base Entry
Facts and rules
”This bank requires the account-opening branch address”
Semantic search or grep exact retrieval
Dedicated Code Tool
Repeatable operational procedures
”API call sequence for querying account balance”
Solidified as code, called via parameters
Skill Document
Complex but frequently changing work strategies
”Best practices for handling insurance claims”
Natural language document, loaded on demand
A simple rule of thumb decides which form to use: purely factual information goes in the knowledge base; frequently used, parameter-heavy procedures become code (tools); frequently changing processes that involve strategic judgment become documents (Skills). The latter two are both “tool generation”—the higher-order form of externalized learning, which externalizes not just knowledge but process into code, trading “rethink it every time” for “generate once, reuse many times”—much like writing an automation script after deploying a server by hand the first time. Chapter 4 covered the framework for choosing between dedicated tools and Skills in detail.
The stance Chapter 1 took toward the Bitter Lesson—endorse the direction, stay pragmatic about the pace—finds its fullest expression in externalized learning. Do not compress all knowledge into parameters, and do not freeze processes into if-else rules; instead, let the Agent actively build an external ecosystem of knowledge and tools, extending the logic of capability growth from inside the model (parameter scale) to the outside world (the scale of tools and knowledge bases). The choice of knowledge carrier follows the same logic: the memories and skills discussed in this chapter mostly settle as Markdown files on a filesystem rather than a hand-designed knowledge graph—the latter is more precise in specialized domains, but natural language is the format models handle best, and with an LLM doing the compression and organizing on top, it forms a general path that depends on no human prior structure and keeps scaling with model capability. Of course, externalized learning itself—in what format to store, how to organize the index, when to distill—still requires engineering design; that is exactly what “pragmatic about the pace” means.
Why Agents Should Learn from Experience: From “Smart” to “Skilled”#
The veteran rep who turned scattered rules into a handbook points at the crux of moving from “smart” to “skilled”: the gap is usually not that the model lacks intelligence, but that much business process and domain knowledge is dynamic and non-public—no amount of general capability in the base model will supply it. These are problems that run on experience, and experience is exactly what an Agent must learn: that canceling a certain service means filling out a specific form, not making a pointless phone call; the eligibility conditions for a promotion (say, veterans, or customers of two-plus years); whether a broadband quote from this carrier in this region still has room to negotiate. Likewise, a Coding Agent doesn’t know a project’s house conventions and deployment process, and a browser Agent doesn’t know a site’s anti-scraping tactics or its latest layout change—all real-time domain knowledge absent from the pre-training data.
With the “why” settled, the question becomes “how.” The engineering practice of externalized learning begins with recording and reusing what worked. The two experiments below demonstrate complementary ways to accumulate experience: one distills high-level strategy into retrievable knowledge summaries (problem-solving notes, in effect), the other freezes concrete operation sequences into replayable automation tools (operation recordings).
Table 8-2 categorizes experience learning mechanisms by layer to help readers understand the relationships between knowledge distillation, knowledge organization, knowledge application, and engineering support.
Table 8-2 Layers of Agent Experience Learning Mechanisms
Extract reusable knowledge from successful and failed experiences
Knowledge Organization
Skills, Sleep Consolidation
Structure and index knowledge for storage
Knowledge Application
System Prompt Optimization
Inject knowledge into the Agent’s behavior pattern
Engineering Support
Cross-Session Continuation
Enable long tasks to execute persistently
These four layers interweave through the rest of the chapter: strategy summaries, workflow recording, and learning from failure (Knowledge Distillation) lead naturally into Skills and sleep consolidation (Knowledge Organization), then system prompt optimization (Knowledge Application), closing with cross-session continuation of long tasks (Engineering Support).
Figure 8-2: Strategy Summary Learning-Application Loop in the GAIA Experiment
Experiment 8-1 ★★: Learning from Successful Experience: Strategy Summary
The gaia-experience project is a typical implementation of the “Strategy Summary” idea. A strategy summary condenses a successful problem-solving process into a structured experience note—recording “what methods were used, what pitfalls were encountered, and what the key steps were”—so that it can be directly referenced when encountering similar problems in the future.
Not every run trajectory deserves to become experience. The criterion is transferability: will the lesson from this task carry over to similar tasks in the future? A fix valid only for one specific input has no place in long-term memory.
This experiment uses two key infrastructures. The AWorld framework is an open-source execution and evaluation environment specifically designed for AI Agents, providing a standardized set of tools (browser, file system, code interpreter, etc.) and an automated evaluation pipeline—think of it as an “exam room” for Agents. GAIA is a highly challenging benchmark that evaluates general-purpose AI Agent capabilities through complex, multi-step problems requiring human-like intelligence—for example, “find specific information on a website, process it with code, and calculate the answer,” often requiring the combined use of a browser, file manager, code interpreter, and complex logical reasoning.
The core innovation is adding a complete “learning-application” loop to the Agent within the AWorld framework. In Learning Mode, whenever the Agent successfully completes a GAIA task, the system automatically captures its complete action trajectory and uses an LLM to “reflect” and “summarize” it, generating a structured experience summary. This summary not only contains the final answer but also distills the core method, key insights, and effective tool sequences used to solve the problem. These experiences are vectorized and stored in a knowledge base. In Apply Experience Mode, when the Agent receives a new task, it first performs a semantic search in the experience knowledge base to find the most similar historical success cases, and injects these experiences as “success examples” into the system prompt to guide decision-making. The experiments show this significantly improves both the efficiency and the success rate on new problems—the more tasks the Agent solves, the richer its experience and the stronger its capabilities: a self-evolving system running on positive feedback.
Experiment 8-2 ★★: Learning from Repetitive Tasks: Workflow Recording and Replay
The browser-use-rpa project is an excellent example of the “Workflow Recording” idea. Workflow recording works like Excel’s macro recorder: record the steps the first time you do something by hand, then repeat them with a single click of “playback.” The problem it solves is very practical: many repetitive browser operations (sending a report email, querying a particular website) change their parameters each time—recipient, search keyword—but keep the same core flow. Making the Agent start from scratch every time, burning an expensive multimodal LLM to “rediscover” that flow, is an enormous waste: it leans entirely on in-context learning and never externalizes success into a reusable tool. At its core, the project is a head-to-head experiment in efficiency and cost, pushed to the extreme.
In the Learning Phase, the Agent performs the task for the first time, completing the operation through the multimodal LLM’s observe-think-act cycle, just like a human. Each time the LLM decides to execute an action, the system extracts the precise positioning information of the target element from the browser-use framework’s history: the webpage is rendered as a DOM tree (Document Object Model) in the browser, where each button, input field, and link is a node; XPath (XML Path Language) points to a specific node using a path-like syntax such as /html/body/div[2]/button[1]. The action is recorded as a structured step: action type (click, input, etc.), XPath of the target element, action parameters, and post-execution verification information (e.g., whether the page URL changed, whether the expected element appeared). After the task is successful, the LLM generates a semantic label (e.g., “Send Email”) and a description (e.g., “recipient field, subject field, content field, send button”), which are stored in the knowledge base along with the step sequence, forming a parameterized “workflow” entry.
In the Replay Phase, when a new task arrives, the system checks for a matching existing workflow using both semantic similarity (embedding vectors) and key element checks. If a match is found, it executes the steps at high speed: it uses Playwright’s (an open-source browser automation library) waiting mechanism (page.locator(xpath).wait_for(state='visible', timeout=15000)) to ensure elements are loaded; parameterized templates (e.g., “Enter {{email}} in the recipient field”) extract actual parameter values from the current task instructions via a lightweight LLM call, without requiring full visual reasoning. If a step fails (element not found, wait timeout), it indicates the webpage structure may have changed. The workflow is then marked as “potentially outdated,” and the system falls back to learning mode, re-completing the task through LLM reasoning and generating a new workflow to replace the old one.
Acceptance Scenario: Sending an email in the Gmail web interface.
First Execution (Learning Phase): “Send an email to test@example.com with subject ‘Test Email’ and body ‘This is a test email.’” Observe how the Agent uses a multimodal LLM to identify the “Compose” button, recipient input field, subject and body input fields, and the “Send” button. Record the operation steps, time taken, and number of LLM calls.
Repeated Execution (Replay Phase): “Send an email to another@example.com with subject ‘Follow-up Test’ and body ‘Second test email.’” The system identifies the matching workflow, extracts the new parameter values, and directly replays the operations without requiring LLM visual reasoning. The time taken and number of calls should be significantly reduced.
Knowledge Update: Simulate a webpage redesign (modify the HTML structure so the XPath of a certain button changes), and verify that the Agent can detect the workflow failure, fall back to learning mode, and regenerate a workflow to update the knowledge base.
Expected observations: Task execution speed during the replay phase is significantly improved (by several times), LLM call costs are drastically reduced, and the success rate is more stable.
Workflow recording is not an isolated engineering trick; behind it stands a more general methodology. Voyager, an open-world Agent architecture from the NVIDIA team (detailed later), systematizes the explore-consolidate cycle in the virtual world of Minecraft: execute the task → verify success → store the successful action sequence in a skill library → retrieve and reuse it on similar tasks. Experiment 8-2 is this playbook applied to browser automation: the learning phase is “exploration,” the workflow knowledge base is the “skill library,” and replay with fallback on failure is “retrieval and reuse” plus continuous improvement.
Experiment 8-2 also exposes the two most fragile links in record-replay; handle them cleanly and the mechanism becomes genuinely reliable1. The first is when to trust the replay. The robust move is to compile a successful action sequence into a small state machine program, where each state carries a “verification predicate”—a UI pattern that must hold on the current, real screen. During replay, the predicate is checked against the live screen before every action: look first, then act. The moment a predicate fails or an action errors out, control returns to the full Agent to redo the task, and the fresh trajectory is compiled into a program again. Because replay needs zero model calls, repeat tasks that hit the cache run 8.5–13 times faster. The second link is don’t store bad programs: immediately after compiling, reset the environment and replay from scratch, using the benchmark’s built-in evaluator to confirm the job actually got done before the program enters the library. This pre-storage verification blocks programs that replay 100% of their steps yet never accomplish the task (the whole flow runs, Save gets clicked—but one field was empty all along). Without the gate, faulty programs accumulate and the library rots. It reduces to one clean principle: procedural memory needs a verification gate, or the self-improvement loop decays—the strict version of Experiment 8-2’s “detect workflow failure, fall back and relearn.”
Strategy summaries and workflow recording both mine successful trajectories—Experiment 8-1 triggers reflection only after a task succeeds. But failures are just as worth keeping, and often carry more information: a failure definitively rules out a path, while a success is merely one viable path among many. Failure experience typically crystallizes into two forms: error pattern libraries (recording which method fails under which circumstances, and what the failure signal looks like) and negative rules (“stop using method X for Y”—for instance, “don’t cancel subscriptions with this carrier by phone; the phone channel has no authority to process them”).
The representative work here is Reflexion (Shinn et al., 2023)2: after a task fails, the Agent reflects on the cause in natural language (“at step three I should have verified identity first, not submitted the form directly”) and stores the reflection in episodic memory. On the next attempt at a similar task, those reflections are read back in as extra context, and the same mistake is not repeated. No model parameter is ever updated—Reflexion is the classic example of evolution without changing weights. A reflection carried in language also holds far more information than a scalar reward, a point we return to when discussing system prompt learning. The other important outlet for failure experience is the system prompt itself: the automatic prompt optimization discussed later in this chapter writes negative rules extracted from failure cases (“never transfer to a human agent over a policy dispute”) into the system prompt, turning them into behavioral constraints that bind every subsequent task.
Skills: Externalizing Domain Knowledge into Structured Capabilities#
The two mechanisms above capture “how to think” and “how to do.” The Skills mechanism takes a third path: systematically refining domain know-how into structured capability modules that load on demand. Think of a Skill as a job manual—a new employee needn’t figure everything out from scratch; read the manual and get to work. Chapter 2 covered Skills’ Progressive Disclosure mechanism (metadata → core process → details) and their KV-Cache-friendly design in detail; this section is about the philosophy of knowledge externalization behind Skills, and how to generate them automatically.
The core value of Skills is carrying knowledge in human-readable text: quick to update (no retraining), auditable (human experts can edit and improve them directly), and portable (they survive a change of model or system). In essence, Skills convert domain knowledge trapped in unstructured documents into a structured form Agents can readily use—letting the Agent exploit knowledge through its general search and reasoning abilities, instead of hardcoding that knowledge into program logic.
Going further, Anthropic’s Skill Creator3 is a meta-capability that can create other Skills. It guides the Agent to refine domain operational knowledge into structured Skills through observation, learning, and summarization. When asked to create a Skill for a specific domain, the Agent first understands the specific usage scenarios through dialogue with the user, then analyzes each scenario to identify reusable resources, and finally creates a complete Skill package containing a standard directory structure, scripts, references, assets, and a main SKILL.md document. Skill Creator enables the knowledge transformation process itself to be completed by the Agent, realizing a bootstrapping cycle of knowledge accumulation: Agents can not only use Skills but also create them.
Claude Code’s CLAUDE.md mechanism shows the same capability: on first contact with a repository, it reads through the codebase and generates a project guide—architecture, coding conventions, how to run the tests—which it then consults and updates throughout later development. Automated Skill generation means an Agent’s growth is no longer bounded by how much time and expertise human experts can spare: entering a new domain, the Agent can explore on its own, build its operating guide, and freeze it into a Skill—moving from “relying on pre-programmed knowledge” to “learning and accumulating knowledge through practice.”
Seen as experience consolidation, tool generation takes two concrete forms: dedicated code tools, and Skills paired with a general executor. The selection principles appeared earlier, and Chapter 4 gives the full framework, so we won’t repeat them—applied to this section: operations with complex parameters and frequent calls become code tools (Voyager’s Minecraft skill library, the parameterized scripts from browser workflow recording), while strategic, volatile business rules become Skill documents (Claude Code’s CLAUDE.md). Real systems usually mix both.
Sleep Learning: Autonomous Evolution of User Memory#
The mechanisms so far—strategy summaries, workflow recording, Skill generation—all operate during task execution or in its immediate aftermath. But human learning has another crucial phase: memory consolidation during sleep. Chapter 2 invoked this analogy for context compression—the brain works the day’s sensory input into compact long-term memory. The analogy stretches beyond a single session to cross-session experience management: the day’s scattered experiences are reorganized during sleep, stripped of redundancy, woven into the existing knowledge network, and emerge as long-term memory that is more compact and easier to recall.
The most natural object of this offline consolidation is the Agent’s memory of the user—who you are, what you prefer, what facts you’ve mentioned. One common misconception is worth dispelling first: what an Agent like Claude Code organizes during “sleep” is primarily user memory, not a shared knowledge base. Knowledge bases (Chapter 3’s RAG) carry domain documents with no tie to any particular user, batch-loaded by offline pipelines and rarely changed. User memory is different—a model of you, accumulated piecemeal across conversations, that comes to know you better over time—and it is exactly this part that needs repeated “sleep consolidation.” Claude Code and Hermes, introduced next, both store this kind of user memory; our focus is how it evolves autonomously.
To be clear about the division of labor with Chapter 3: that chapter covered how user memory is stored and queried, along with the storage layer’s consolidation algorithms (clustering summaries, conflict versioning, and so on), none of which we repeat. This section is about the engineering and evolution questions—when to consolidate, who does the consolidating, and what form the results take—so that memory grows more accurate with use.
Claude Code: Storing User Memory in Markdown. Claude Code stores user memory as plain, human-readable Markdown: each memory is a small file with frontmatter metadata recording exactly one fact, and an index file (MEMORY.md) provides summary navigation. The benefits are plain to see—quick to update (edit the file, no retraining), auditable (users can open and change it directly), and portable (it survives a change of model or system).
But recording is not enough; the memory also needs organizing. Claude Code engineers the sleep-consolidation metaphor into a background mechanism that runs periodically. (The description below is based on public-version behavior and community analysis, not an official specification.) The core design idea: accumulating experience and consolidating memory are two independent processes that should not share a time window—Agents, too, need dedicated review time. Concretely, when two gating conditions hold (enough time since the last consolidation, and enough new sessions accumulated in between), the system launches an independent sub-agent in the background to run a four-stage consolidation: Orient (read the existing memory index to grasp the overall knowledge landscape), Gather (search recent sessions for new information worth persisting, and detect facts that contradict existing memory), Consolidate (merge new signals into existing topic files rather than spawning near-duplicates, convert relative dates to absolute ones, delete old facts that have been disproven), and Prune & Index (cap the index size, drop stale pointers).
The mechanism’s most important design decision: consolidation never runs during user interaction—it completes asynchronously in the background, invisible to the user. Dual gating and distributed locks keep concurrent instances from triggering it twice; failures roll back automatically and retry next time; the consolidation sub-agent’s permissions are confined strictly to the memory directory. Step back, and this is user memory management evolving from “record but never organize” into a full record—consolidate—prune lifecycle. Without regular consolidation, the memory store degrades into a low signal-to-noise dump that drags retrieval down; with it, the store stays compact, consistent, and navigable—just as a human expert’s knowledge is not an endless pile of facts but a structured understanding, refined through repeated reorganization.
Hermes: Making Autonomous Learning a Resident Service. Nous Research’s open-source Hermes (2026) takes the idea to its logical conclusion: a daemon resident on the user’s own machine, accumulating memory and evolving autonomously across sessions. Its memory is divided into four layers4: Prompt Memory (MEMORY.md and USER.md, injected at session start, deliberately limited to a few thousand characters to “force” the Agent to prioritize), Session Retrieval (using SQLite full-text index FTS5 for historical sessions, retrieved fragments are first summarized by an LLM before injection, bringing in only parts relevant to the current task), Skill Library (procedural memory, using progressive disclosure, loading only skill names and summaries by default), and an optional Honcho User Modeling Layer (passively tracks preferences, communication style, and domain knowledge in the background, portraying “how the user and Agent co-evolve” across sessions). When a task meets certain conditions (more than five tool calls, recovery from an error, a user correction, or a non-obvious workflow carried through), Hermes automatically solidifies the experience into a reusable skill, preferring incremental patches over wholesale rewrites. Claude Code and Hermes represent today’s mainstream form of autonomous user memory evolution—both carried in human-readable Markdown and text.
The mechanisms so far all deposit experience and memory outside the model—knowledge bases, workflows, Skill files, user memory. But there is one more carrier of experience, and it is the most direct of all: the system prompt itself.
Andrej Karpathy argues that current LLM training is missing an entire learning paradigm: “system prompt learning.” Pre-training acquires knowledge; fine-tuning instills habitual behavior; both change model parameters. Yet much of human learning looks more like updating a system prompt—when we puzzle something out, we write it down for ourselves in explicit language: “next time I hit this kind of problem, try this method first.”
LLMs, Karpathy observes, are like the protagonist of Memento—waking each time with no memory of what came before, and we haven’t even given them a notebook. Reading Claude’s system prompt (roughly 17,000 words, varying by version), he found it packed with general problem-solving strategies, such as: “If asked to count words, letters, and characters, Claude should think step-by-step before answering, explicitly counting by assigning a number to each letter.” That one exists to handle questions like “how many ‘r’s are in ‘strawberry’?”
Knowledge of this kind, Karpathy argues, shouldn’t be hand-crafted by humans—it should come from system prompt learning. The idea shares ground with reinforcement learning: both use failures to improve future behavior. But the learning algorithms differ—system prompt learning edits the prompt text directly, while reinforcement learning adjusts parameters by gradient descent—and the former is dramatically more data-efficient, because the feedback channel has more “dimensions.” This is Karpathy’s critique of outcome-based reinforcement learning: a single scalar reward (“correct/incorrect”) carries far less bandwidth than a full natural-language post-mortem (“you should have verified the ID before starting the refund process”). From the very same failure, system prompt learning absorbs far more than one bit.
In my view, the essence of system prompt learning is sharpening rule boundaries through edge cases. Most rules work fine in typical scenarios; the real challenge is the gray zone. “Transfer to a human agent when the request exceeds your capabilities” sounds clear enough—but does a user unhappy with policy count as exceeding capabilities? What about a user demanding an exception? It is these edge cases that define what a rule actually means.
Where reinforcement learning must grind through massive trial and error to move the weights, system prompt learning learns from one or a handful of edge cases. Hit a failure, add a clear rule to the prompt immediately—no need to collect thousands of similar samples for fine-tuning. The learning is not just data-efficient but fully interpretable: every rule is written out in plain text, and can be audited, amended, or deleted. As edge cases accumulate, the system prompt evolves into a detailed problem-solving handbook, the way an expert keeps refining her notes on the job.
How do we automate this? The key is to bring in a Coding Agent. System prompts and tool descriptions are themselves documents and code, scattered across files. When an edge case surfaces, the Coding Agent can (1) read and understand the existing prompt, analyzing the rule structure and the failure’s context; (2) generate a precise, code-level diff—which file, which location, what change; and (3) maintain consistency, ensuring the new rule introduces no contradiction or redundancy. Final say stays with human experts, who review each diff and judge whether it is sound.
Automated prompt optimization is not just Karpathy’s idea; it’s a well-established research area in academia. DSPy5 treats prompts as optimizable parameters of a program: developers only declare “what goes in and what comes out” for each module, and the framework automatically searches for example combinations and instruction phrasing on an evaluation set, transforming prompt engineering from manual debugging to systematic optimization. OPRO6 lets the LLM itself act as the optimizer: using historical prompts and their scores as context, the model iteratively proposes better rewrites, outperforming human-designed prompts on tasks like mathematical reasoning. GEPA7, proposed in 2025, goes further: it performs natural language reflection on failure trajectories, evolves prompts accordingly, and maintains a Pareto frontier among multiple candidates (i.e., a set of candidates, each with unique strengths, where none can be fully surpassed by another, rather than keeping only a single “optimal” solution) to preserve complementary optimization directions—outperforming GRPO fine-tuning (introduced in Chapter 7) on multiple tasks while requiring one to two orders of magnitude fewer samples. GEPA is precisely what this section calls “system prompt learning,” and its empirical results support the earlier judgment about feedback information volume.
These automated frameworks differ from the “Coding Agent writes diffs + human review” approach on three counts. First, offline versus online: the frameworks batch-optimize against offline evaluation sets, while the diff approach evolves case by case as edge cases arrive in production. Second, human oversight: the frameworks rewrite end-to-end—efficient, but liable to produce odd phrasing that overfits the evaluation set—whereas the diff approach keeps a human in review, so every rule stays interpretable and accountable, a better fit for high-stakes settings like customer service. Third, the evaluation set itself: DSPy, OPRO, and GEPA all need scored task sets to drive their search; the diff approach needs one failure case and a line of human feedback. In practice they complement each other: batch-optimize the initial prompt with an automated framework, then let the diff approach carry the continuous evolution after launch.
Experiment 8-3 ★★: Automatic Optimization of System Prompts
Experiment Goal: Implement an automated system prompt learning mechanism based on human feedback.
Technical Approach: Design a system prompt learning workflow based on the tau-bench airline customer service scenario. The initial Agent’s human transfer rule is “Transfer only when the request cannot be handled within your action scope.” Evaluation reveals the Agent over-transfers—immediately transferring to a human upon encountering a policy dispute instead of trying to explain the policy to the user. Human expert feedback indicates that policy disputes should be handled by patiently explaining the policy, not by transferring. The Coding Agent reads the system prompt file, locates the relevant rule, and generates a precise modification: clarifying the transfer boundary as “user explicitly requests a human agent + emergency safety situations,” adding a negative rule “never transfer due to a policy dispute,” and implementing code-level changes.
Control Group: Manually tuned system prompts (without the automated optimization process).
Expected Observation/Acceptance Criteria: The optimized system prompts show no performance degradation on the original retained task set (new rules do not break existing correct behaviors), while accuracy improves on the set of edge cases that trigger over-transfer—i.e., for policy disputes, the Agent no longer immediately transfers but first attempts to explain the policy.
Cross-Session Continuation of Long Tasks (an Engineering-Support Aside)#
Strictly speaking, this section covers not an experience distillation mechanism but the engineering support for self-evolution (Table 8-2’s Engineering Support layer): applying externalization to task state, so that both what has been learned and what is half-finished survive across sessions. It descends directly from the Coding Agent workflow of Chapter 5 and belongs here because it leans on the same core move—writing state outside the model. Many tasks (building a complete application from scratch, say) dwarf a single session’s context window. Even with context compression on, two failure modes remain: try to finish the whole application in one session and the context runs out first; finish only part of it, and the next session cannot accurately restore progress—so it declares the task done too early.
The more stable approach splits a long task between two roles, an Initializer Agent and a Coding Agent—a project manager who breaks the work down and writes the checklist, then an engineer who works through it item by item. The Initializer Agent runs exactly once, in the first round: it generates a structured feature list (say, feature-list.json), an initialization script, an initial git commit, and a progress file (progress.json), turning the task into persistent file system state. Every later session is the Coding Agent running one loop iteration: restore context from the progress file and git log, locate the next feature to implement, implement it and run the tests, update the passes field, commit, exit. The key constraints: progress lives in files, not in the context; the feature list is JSON, not Markdown (structured formats are more stable for a model to read and write); and the task counts as complete only when every feature reads passes: true. A mid-run crash then costs nothing—the task resumes straight from the file system. Once a task runs past half an hour, crash recovery stops being optional and becomes mandatory.
Chapter 4’s “Proactive Tool Discovery” solves the problem of finding the right tool among those that exist. The next capability up—the subject of this chapter—is harder: what if the tool you need doesn’t exist at all? How does an Agent discover and create new tools on its own?
The Fundamental Limitation of Predefined Tool Sets#
Most current AI Agent systems rest on an implicit assumption: that a sufficiently complete tool set can be defined in advance to cover the vast majority of tasks. In a closed domain this may hold—a dedicated customer service Agent might need only a dozen tools. For a truly general-purpose Agent, the assumption is wishful thinking.
The fundamental difficulty: the tools the real world demands are nearly infinite and cannot be enumerated up front. Even when the library holds something similar, its interface and parameters rarely match the need at hand exactly—so it runs inefficiently or breaks. And a vast number of useful services simply don’t exist behind Agent-friendly standard interfaces; each one costs a round of manual development to adapt. In the end, the predefined paradigm caps the Agent’s capabilities at whatever its human engineers managed to foresee and prepare.
Breaking this limitation takes a fundamental shift: promote the Agent from tool user to tool creator. Instead of waiting passively for humans to hand it tools, the Agent goes out into the open world to find, learn, adapt, and create them as the task demands—the second dimension of self-evolution named at the start of this chapter.
The core idea is to give the Agent a minimal foundation and let it expand its own capability boundary through interaction with the environment and use of external resources. As the Alita paper 8 puts it—“Minimal Predefinition, Maximal Self-Evolution”: a small set of carefully designed foundational tools supplies the basic ability to interact with the world, and the self-evolution mechanism builds unlimited expansion on top of it.
Self-evolution does not deny the value of predefined tools; it builds a layered capability system on top of them.
The key to the shift is turning the global open-source ecosystem into the Agent’s resource pool. Facing a new task, it doesn’t wait for humans to prepare tools—it searches out the library that best fits the need and writes glue code to wire things together where necessary. Tools once used are kept, so the next similar task calls them directly instead of reinventing the wheel.
Agent Autonomously Finds and Executes Tools from the Web#
Figure 8-3: Agent Self-Evolution Pipeline (from Need Identification to Tool Registration)
MCP (Model Context Protocol, detailed in Chapter 4) is the standard protocol by which Agents discover and call tools—think of it as the communication spec for a tool marketplace, through which the Agent browses what’s available, understands input/output formats, and initiates calls reliably. Take the Alita system on a concrete task: “In a YouTube 360 VR video released in March 2018, narrated by the voice actor for Gollum from The Lord of the Rings, what number does the narrator mention directly after first showing a dinosaur?” This demands a specific domain capability—extracting and analyzing video content. Rather than report “cannot complete,” the Agent launches a multi-stage self-evolution process:
MCP Brainstorming Stage: Analyze the task requirements, identifying the need for a “YouTube video subtitle scraper.” The specific implementation involves having the LLM analyze the task requirements and generate a set of candidate tool descriptions (e.g., “a tool that can fetch YouTube subtitles”), then search the MCP server registry for matching existing servers or mark them as needing to be created.
Web Agent Execution Stage: Search open-source repositories, finding the Python library youtube-transcript-api (GitHub: jdepoix/youtube-transcript-api).
Manager Agent Synthesis Stage: Visit the GitHub repository, read the README and code examples, understand the core API, and deduce the environment configuration and code framework.
Manager Agent Execution Stage: Encapsulate the learned knowledge into a tool conforming to the MCP protocol, extract the target video’s subtitles, analyze the content, and find the answer “100000000.”
The newly created tool is saved to the tool library, ready for direct reuse when encountering similar video analysis tasks in the future.
Integrating existing tools from the open-source ecosystem is the first mode, but not every need has a ready-made answer. The Agent must also show a second capability: writing code from scratch to create new tools.
As defined at the chapter’s start, tool generation is the higher-order form of externalized learning—and here it goes one step further, externalizing the “process” itself into precisely executable code.
The key difference is where the code ends up. In a traditional Agent system, code execution is one-off: the Python interpreter runs a script for the current task, and the code is thrown away. In the self-evolution paradigm, the Agent writes code to create reusable, modular tools, saved for good in the tool library, no longer dependent on the model’s transient context or parametric memory. Two benefits follow: experience stops evaporating at the end of each session and accumulates permanently, and code tools behave deterministically and testably—far more reliable than making the model re-derive the solution every time.
The creation process itself follows the familiar rhythms of software engineering: requirements and interface design, algorithm selection and implementation, testing and validation, and finally generating an MCP-conformant schema and registering it in the tool library. Relative to human engineers, Agents err more often at understanding requirements, debugging by intuition, and pinning down edge conditions—so production systems pour the bulk of their compute into testing and validation, using massed automated tests to soak up the uncertainty from the earlier steps.
Voyager: An Agent Continuously Learning in a Virtual World#
We have seen how an Agent can discover and create tools on its own. Voyager pushes the idea to its limit inside the virtual world of Minecraft: it doesn’t just use tools—it builds a reusable skill library out of its successes, achieving self-evolution in the truest sense: the more it practices, the more skilled it becomes.
Voyager is externalized learning at work in an open world: a Minecraft Agent that learns continuously by distilling every successful experience into executable code tools and storing them outside itself.
Its architecture rests on three key elements:
The automatic curriculum generator proposes the next exploration goal (“find iron ore,” “craft an iron pickaxe”) from the Agent’s current state, mastered skills, and surroundings. Each goal sits in the Agent’s zone of proximal development—neither too easy nor too hard, like well-paced level design in a game.
The skill library is the core mechanism. When a new task succeeds, its action sequence is distilled into executable code and stored for good. Skills are hierarchical and composable: “craft a wooden pickaxe” calls foundational skills like “chop a tree” and “craft wooden planks.” A new task is often solved quickly by retrieving and combining existing skills, sparing the LLM from thinking everything through from scratch.
The iterative prompting mechanism keeps skills improving. On failure, feedback is gathered—environmental observations, error messages, self-verification results—folded into the LLM prompt to guide improved code, and iterated until stable.
As Voyager explores Minecraft, its skill library keeps growing: the paper reports it unlocks the technology tree’s key milestones (wood, stone, iron, diamond) significantly faster and discovers 3.3 times as many unique items as baseline methods. At bottom, this continuous learning is externalized learning making the leap from temporary adaptation to permanent accumulation—every successful exploration becomes a reusable code tool. Voyager proves the paradigm works and hands us a complete methodological blueprint for building self-evolving Agents in the real world—which the following experiment does, in a real-world tool discovery setting.
Experiment 8-4 ★★★: Agent Finds Tools from the Web to Achieve Self-Evolution
Basic Tool Configuration: Only web_search, read_webpage, code_interpreter, create_tool, search_tools. No domain-specific predefined tools.
Task One: YouTube Video Content Understanding—“In a YouTube 360 VR video released in March 2018, narrated by the voice actor for Gollum from The Lord of the Rings, what number does the narrator mention directly after first showing a dinosaur?” Expected Process: The Agent analyzes the task, identifies the need for the ability to extract YouTube subtitles; finds the youtube-transcript-api library and its GitHub repository via search; reads the README and documentation to understand the installation method and interface; tests the library using the code interpreter; writes wrapper code to package the subtitle extraction functionality as a standard tool; uses the new tool to extract the target video’s subtitles and analyze the content. Success Criterion: Output the correct answer “100000000.”
Task Two: Real-Time Financial Data Query—“As of today, what is the latest stock price of NVIDIA (NVDA)? What is the percentage change compared to one week ago?” Expected Process: The Agent identifies the need for real-time stock data capabilities; searches for available financial data APIs (yfinance, Alpha Vantage, etc.); evaluates multiple options based on ease of use, need for API keys, and data completeness; selects the most suitable option and creates a query tool. If a certain API requires registration that cannot be completed automatically, the Agent should switch to a backup plan rather than hallucinating or giving up directly.
Verify Tool Reuse: When executing a similar type of task again, the Agent should directly retrieve the already created tool from the tool library, rather than going through the search and creation process again.
Challenges and Difficulties: Search precision (may find unsuitable libraries or outdated information), documentation understanding (some open-source projects have incomplete documentation, requiring synthesis from multiple sources), environment configuration (some libraries have complex dependencies or system requirements), error recovery (the first attempt may fail, requiring the Agent to debug and correct), hallucination control (must work based on actual search results and documentation, cannot assume the existence of a library or the usage of an API out of thin air).
Continuous Accumulation of Three-Layer Capabilities#
Continuous learning shows up at three levels:
Tool level: Successfully created tools go into the tool library, and new tools can build on existing ones to form a hierarchy—once a “get stock price” tool exists, a “portfolio analysis” tool can stand on it.
Knowledge level: Every tool created brings knowledge with it. The Agent gradually learns which open-source libraries suit which tasks, which APIs work without applying for a key, which libraries keep fighting the system environment. Extracted as heuristic rules or case libraries and stored in the knowledge base (storage and retrieval in Chapter 3), this knowledge guides future tool creation.
Strategy level: Through repeated practice, the Agent improves its self-evolution strategy itself. Early on it may pick the wrong library, write overwrought logic, or miss critical edge cases; fed by both failures and successes, it learns to judge open-source project quality more accurately, implement more concisely, and test more thoroughly. This meta-learning means the Agent’s capacity for self-evolution is itself evolving. In the short run, such meta-experience accumulates in system prompts and Skills; in the long run, once stable, it can be baked into weights by reinforcement learning (see Chapter 7)—which lies beyond this chapter’s “no weight changes” scope.
The first two levels are externalized learning applied directly—tools accumulate in the tool library (this chapter), knowledge in the knowledge base (Chapter 3). The strategy level shows the relay between externalized learning and post-training: iterate fast on interpretable, editable external carriers first, and only once the strategy stabilizes consider freezing it into parameters (Chapter 7).
Self-evolution gives Agents formidable room to grow—and a set of safety risks all their own.
Supply chain attack is the most direct threat: an Agent that searches for and installs open-source libraries from the internet may automatically download and execute a malicious PyPI or npm package. It is like letting a new employee download whatever software they like—without constraints, compromise is only a matter of time. Mitigations include running tools in a sandbox and automatically security-scanning newly created tools.
Capability drift is more insidious: the strategies and tools an Agent accumulates through continuous learning can slide away from the designer’s original intent, especially over long unsupervised runs. Countermeasures include a whitelist of permitted tool types, limits on how far the tool library may grow, and periodic human review of new tools.
Tool quality degradation deserves attention too: an auto-generated tool that lacks sufficient testing can produce wrong results at the edges, and reuse propagates those errors into later tasks—a buggy tool called over and over does far more damage than a one-time inference mistake.
Memory and experience poisoning is the attack surface most intrinsic to a mechanism that writes to itself. Content the Agent touches during a task—web page text, tool outputs—may carry maliciously injected instructions (prompt injection, detailed in Chapters 2 and 4). Let that content settle into long-term memory or Skills as “experience” without review, and the attack turns from one-shot into persistent: a contaminated entry stays live across sessions, striking every time it is retrieved. Compared with in-session prompt injection, this is stealthier and harder to root out—the victim is not one response but the Agent’s “knowledge” itself. Mitigation runs at three levels: pre-write review and source tagging (record which task and which source each experience came from, and run injection detection on untrusted content before it is stored); channel isolation between experience and instructions (inject retrieved experience into context as reference material, not commands, telling the model explicitly that experience carries no directive authority); and injection detection at retrieval (lightly scan retrieved entries for injection patterns, downgrading or alerting on anything suspicious). Knowledge freshness and poison defense are two sides of one coin—freshness fights natural obsolescence, poison defense fights malicious contamination—and both demand that the experience library trace sources and be able to evict entries. (How a freshness mechanism detects and retires outdated experience is left as an extension of Thought Question 3.)
Experiment 8-5 ★★★: Design an Evaluation Dataset for Self-Evolving Agents
The preceding experiments demonstrate how Agents learn from experience, discover tools, and create tools. But how do we evaluate these self-evolution capabilities? This requires specially designed evaluation datasets—ones that test both tool discovery and creation abilities while avoiding simple memorization of fixed tool usage patterns.
Construct 20 tool-requiring tasks across different domains (multimedia processing, financial data, scientific computing, geographic information, social media, IoT device control, etc.). Task descriptions should only state the goal without hinting at tool names—for example, “Get the subtitles of a YouTube video” rather than “Use youtube-transcript-api,” or “Query real-time cryptocurrency price trends” rather than “Use the CoinGecko API”—ensuring the Agent must genuinely search for or create tools. The basic tool set typically includes: web search, web page reading, code interpreter, and tool creation and registration.
Design a four-layer hierarchical validation:
Task Correctness: Subtitle content is accurate, financial data matches reality
Tool Discovery Effectiveness: Analyze search keywords, visited web pages, and selected libraries to judge
Tool Creation Quality: Error handling, parameter validation, documentation completeness, scored by LLM-as-a-Judge using a Rubric
Tool Reuse Capability: Whether the second execution of a similar task directly retrieves the created tool instead of repeating the search
Prepare reference solutions (recommended open-source libraries and typical API examples) and known pitfalls (deprecated libraries, APIs requiring paid registration, etc.) for each task.
This chapter has laid out the methodology by which an Agent keeps expanding its capabilities without modifying model weights—self-evolution.
Building on Chapters 1 and 7, we first placed self-evolution among the three learning paradigms: post-training freezes strategy into parameters (detailed in Chapter 7, cited here only for contrast), in-context learning handles temporary adaptation, and this chapter’s subject is the path that leaves the weights alone—externalized learning and its extensions. From there we moved into engineering practice.
The chapter unfolded along self-evolution’s two dimensions. The first is distilling knowledge from experience: strategy summaries capture decision-making wisdom, workflow recording freezes operating procedures, Reflexion-style reflection preserves failure lessons, Skills structure domain knowledge, and system prompt optimization extracts rules from edge cases—together, a machinery by which practice makes the Agent proficient. The second is proactively breaking through tool boundaries: building on Chapter 4’s “Proactive Tool Discovery,” which finds the right tool among those that exist, the Agent goes further—searching the web and integrating open-source libraries, writing new tools from scratch—going from tool user to tool creator, with Voyager supplying the complete blueprint for this paradigm in open worlds.
This completes our treatment of the Agent’s core capability system—context engineering, tool design, code generation, evaluation methodology, model post-training, and self-evolution. The next two chapters shift to frontier applications. The next chapter examines how Agents move beyond pure text I/O into multimodal perception and real-time interaction: voice dialogue, Computer Use (GUI automation), and robotic manipulation. These scenarios share two core challenges—multimodality and real-time performance—and it is precisely these that are driving Agent architectures through a fundamental evolution, from serial pipelines to end-to-end models.
★★ The ability of Agents to autonomously search for and integrate open-source libraries (self-evolution) is extremely powerful but also introduces supply chain security risks. A malicious PyPI package could be automatically installed and executed by the Agent. How would you mitigate this risk?
★★ Voyager-style experience learning allows Agents to encode successful tool usage patterns into reusable Skills. However, as the Skill library grows, retrieving the correct Skill itself becomes a new challenge. Does this constitute a recursive problem—does an Agent need a “Skill Retrieval Agent” to manage its own Skills?
★★★ Externalized learning encodes successful experiences into strategy summaries or workflow scripts. But the definition of “success” may change over time (e.g., business rule updates). Outdated experiences are not only useless but potentially harmful. What kind of “knowledge freshness” mechanism would you design to detect and eliminate outdated experiences?
★★★ Workflow recording converts successful operation sequences into replayable automation tools. However, APIs update and UIs change, so a recorded workflow can break at any time. How can workflows be made to automatically repair themselves when the environment changes, or at least avoid hallucinating that the task is complete when errors occur?
★★★ The three learning paradigms (post-training, in-context learning, externalized learning) correspond to three knowledge carriers: model parameters, context windows, and external storage. Which would you use for a business rule that must go live urgently? For a customer service bot whose replies are too verbose? For making generated PPTs match the company’s existing slide style as closely as possible? Do these choices depend on the capabilities of the model you are using?
★★ When an Agent searches for tools on the internet, how does it determine whether an open-source library is “good to use”? Can the Agent learn to evaluate the quality of open-source projects on its own?
★★ This chapter positions Agent self-evolution as a path to “become stronger without modifying parameters.” However, Chapter 7 points out that improvements at the strategy level ultimately require reinforcement learning to solidify. Where is the boundary between these two paths—what kind of capability improvements are suitable for externalization, and what kind are suitable for writing into parameters?
The complete mechanism of compiling successful trajectories into state machine programs with verification predicates and setting a “pre-storage verification” gate is detailed in Li, Bojie. PreAct: Computer-Using Agents that Get Faster on Repeated Tasks. arXiv<2606>.17929, 2026. ↩
Shinn, N., et al. Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv<2303>.11366, 2023. ↩
