Skip to content
NewsletterOct 29, 2026

← Past issues

An Agent Creates an Agent

Sent Oct 29, 2026

AI That Evolves on Its Own, From Design to Learning—Success Rate in Real-World Tasks Still Stands at 33%

When we talk about AI agents, we typically think of a system where the user sets a goal, and the model performs a series of steps using tools such as search, file reading, code execution, and browser manipulation. However, there are still quite a few areas that require human design. Decisions regarding agent architecture and workflows—such as which agents to use, which tools to connect, the order of execution, and which step to revert to in case of failure—were still often predetermined by humans. So, is the next step in work automation not simply “AI doing the work for us,” but rather “AI designing and improving how it works on its own”?

💡 This Week’s Newsletter Summary

  • Designing Agents: We explore how ADAS and AutoAgent automate the agent architecture and workflow itself.
  • Self-Improving Agents: This section compares how MUSE and Agent Lightning gain experience, learn, and improve their performance.
  • There is still a significant gap in real-world tasks: Even Claude Sonnet 4.6, which achieved the highest performance on ClawBench, had a success rate of only 33.3% in actual web tasks.

From the stage where humans designed agents to the stage where agents design agents

One of the key starting points for this trend is “Automated Design of Agentic Systems” (ADAS), presented at ICLR 2025. In previous agent research, humans often designed roles, prompts, tools, and inter-agent connection structures before working to improve performance. ADAS takes this a step further by enabling AI to explore the question, “Which agent structure works best?” on its own. For example, instead of a human directly determining the prompts, how tools are used, and the roles and execution order of multiple agents, a Meta-Agent generates multiple agent codes by combining these elements in different ways and compares their actual performance to identify the best structure.

In the Meta Agent Search proposed by the research team, the Meta-Agent generates new agents as code, evaluates their performance on real-world tasks, and stores agents with good results in an archive. The Meta-Agent then proposes new architectures by referencing previously discovered designs. In other words, this creates an iterative search process: “agent generation → evaluation → archiving → generation of the next agent.” The research team evaluated this approach on ARC, as well as in reading comprehension, math, science, and multi-task problem-solving. ARC (Abstraction and Reasoning Corpus) is a reasoning benchmark in which agents must identify hidden transformation rules by observing several examples of colored grids and then predict the output for a new input. The researchers verified whether the agents discovered in this evaluation could be transferred to other domains and models.

Meta-Agent Search, in which a Meta-Agent creates and evaluates new agents and accumulates previous designs in an archive | Source: Hu et al., Automated Design of Agentic Systems (ICLR 2025), Fig. 1

AutoAgent, presented in the 2026 ACL Findings, extends this trend in a slightly different direction. While ADAS focuses on automatically exploring better agent architectures, AutoAgent aims to provide a zero-code environment where users can create and deploy agents using only natural language. Internally, it dynamically generates or modifies tools, agents, and workflows; the research team describes this as a single Agent Operating System.

In other words, the level of abstraction in task automation is increasing. Whereas in the past, people implemented workflows through code, the trend now is for natural language requests—such as “Create an agent to handle this task”—to serve as the input for system design.

Even an agent that has already been created is no longer fixed.

The next question that arises regarding design automation is, “Where does the agent’s actual work experience go?” While a typical agent can use previous tool calls or observation results as context within a single task, that experience does not influence how the agent behaves in the next task once the current one is complete.

The MUSE paper presented at ACL 2026 Findings addresses this issue from the perspective of “Self-Evolving Agents.” The core of MUSE is autonomous post-execution critique. After each subtask is completed, it analyzes the execution log to generate structured insights into successful strategies and causes of failure, which are then stored in a hierarchical memory. Later, in long-term workflows, this memory is retrieved and utilized for orchestration. This approach allows the agent’s behavior to be gradually modified using the experience accumulated during execution, without the need for separate model parameter updates.

The research team evaluated MUSE using TAC, a long-term task benchmark. TAC divides tasks into multiple intermediate checkpoints and uses a “partial completion score” that reflects not only whether the task was ultimately completed but also the extent to which intermediate steps were completed. When comparing only difficult tasks running on the same Gemini-2.5 Flash-based system, the memory-less version of MUSE achieved a score of 23.65%, while the version with memory achieved 33.41%. This suggests that the architecture itself—which stores execution experience in memory—boosted performance.

MUSE, which performs long-term tasks by switching between multiple applications and continuously improves by accumulating execution experience | Source: Yang et al., MUSE (ACL Findings 2026), Fig. 1

The term “learning” used here is slightly different from “training,” which refers to updating model parameters. MUSE is closer to a nonparametric approach that accumulates experience in memory at test time to improve behavior. So, can the model itself continue to learn within the agent’s environment?

Going Beyond Stacking Memories: Training Real Agents with RL

Agent Lightning v1.0, released in August 2026, builds on this foundation. In traditional Agentic RL, it was often necessary to reimplement the agent’s interaction loop within the RL framework itself. However, in real-world coding agents or general-purpose agents, there is a separate Agent Harness that handles context management, tool protocols, and orchestration; reimplementing this within the training framework can lead to discrepancies between the actual deployment environment and the training environment.

In Harnessed Agentic RL, as proposed by Agent Lightning, the Harness used in actual deployment is directly incorporated into the training process. By placing an OpenAI-compatible LLM proxy between the agent and the model and observing model calls to construct rollouts and training samples, RL can be integrated without significantly modifying the existing Harness code.

**📌Agent Harness: ** An execution environment that wraps around the agent and handles context management, tool invocation, execution order, and more.

The difference between traditional Agentic RL and Harnessed Agentic RL, in which actual agent harnesses participate directly in training | Source: He et al., Agent Lightning v1.0 (2026), Fig. 1

In the experiment, we used Qwen3.5-9B as the base model for reinforcement learning and configured the mini-SWE-agent as a coding agent harness responsible for repository exploration, code modification, and test execution. After training Qwen3.5-9B with approximately 6,000 training samples, the Pass@1 score on SWE-bench Verified increased by 14.6 percentage points, from 41.8% to 56.4%.

When you look at MUSE and Agent Lightning side by side, you can see two distinct approaches to research on self-evolving agents: one that keeps the model unchanged while accumulating experience as memory and skills, and another that updates the model itself through reinforcement learning based on its actual execution trajectories.

But how well would they actually perform if they were given real work to do?

The research conducted so far has generally focused on how to better design agents, how they can learn from experience, and how to train them. So, if we were to deploy today’s agents in a real-world work environment, to what extent could we entrust them with tasks?

A study that tested this question in a real-world web environment is ClawBench, published in the 2026 Findings of EMNLP. ClawBench deployed AI agents not on static benchmark sites, but on actual, live websites.

We organized 153 recurring tasks from daily life and work—such as shopping, making reservations, and applying for jobs—across 144 live platforms and 15 categories. These tasks include those that require users to find information in documents and fill out multi-step forms, or navigate between different pages to complete a workflow. To prevent any real-world adverse effects, only the final submission request was blocked by the interception layer.

ClawBench, designed to perform everyday online tasks on a real production website | Source: Zhang et al., ClawBench (Findings of EMNLP 2026)

After evaluating eight frontier models, even Claude Sonnet 4.6—which demonstrated the highest performance—completed only 33.3% of all tasks.

What’s interesting is that ClawBench’s tasks themselves don’t seem like cutting-edge research problems. Purchasing a product, entering reservation details, and submitting a job application are ordinary tasks that people perform on the web every day. However, from the agent’s perspective, they must locate accurate information in user documents, interpret dynamically changing web UIs, maintain the previous state throughout a long workflow, and fill in multiple fields without errors.

Ultimately, achieving high performance on a specific reasoning benchmark is not yet the same as reliably completing actual business workflows.

The scope of business automation is changing.

When viewed as part of a single trend, these studies reveal how the scope of automation in Agentic AI is expanding.

In other words, the scope of automation is expanding from task execution to agent design, experience refinement, and agent training.

However, as ClawBench demonstrates, increased autonomy does not automatically mean that reliable task automation has been achieved. In fact, the more agents design and learn workflows on their own, the more critical these design questions become: which objectives to optimize, which tools and data to access, how to evaluate failures, and at which stages to incorporate a “human-in-the-loop.”

We still have a long way to go in real-world applications. However, it seems clear that the benchmark for agent research is shifting from simply “Can it perform this task?” to “To what extent can it design, experience, and improve itself with minimal human intervention?”


The next step in AI automation may go beyond automating tasks themselves to automating “how tasks are automated.”

Leave feedback on this issue