Agents in the Real World: From Exploration to Execution

1. Summary and Result

General-purpose agents can write robot control programs, learn from physical feedback, and guide execution online. Their capabilities and costs depend on the task, implementation, and human support.

We studied Astra and Fable 5.1 on block stacking, hanoi tower sorting, and Ethernet insertion using a dual-arm robot platform. Both completed the stacking tasks, but their insertion results differed substantially. Earlier first success did not necessarily mean faster repeated execution.

Online model participation helps with local decisions; program-driven execution reduces repeated-run costs. Human support remains important for directing exploration, preserving experience, and supplying physical information.

Exploration

Time to first success · Compare approaches ↗

Block stacking

Astra~80 min

Fable 5.1^*~40 min

Hanoi tower sorting

Astra^*~1 day

Fable 5.1~7 days

Ethernet insertion

Astra^*~1.5 days

Fable 5.1No success within 10 days

Execution

Average execution time · Compare results ↗

Block stacking

Astra~5.8 min

Fable 5.1^*~1.1 min

Hanoi tower sorting

Astra~46.5 min

Fable 5.1^*~7.3 min

Ethernet insertion

Astra^*~22.2 min

30 selected recordings across three tasks. Exploration times are operator-reported under unmatched development conditions; Ethernet success denotes mechanical insertion. Compare task results ↗

2. Introduction

Controlling real robots through software tools requires handling calibration errors, changing contact conditions, sensor biases, and execution delays. Recovering from a failed attempt also carries a practical cost.

We investigated four questions about general-purpose agents under these conditions:

• What kind of manipulation tasks can they complete in the real world?
• How do they explore and develop working solutions?
• How efficiently do those solutions execute in repeated runs?
• What role does human support play in exploration and execution?

Our study therefore examines two stages: developing a working solution and executing it repeatedly. Task completion provides the outcome; exploration records, implementations, and human interactions help explain it.

3. System and Evaluation Setup

Hardware platform

The platform consists of two Franka arms, two Robotiq grippers, two wrist-mounted RealSense D405 cameras, and one ZED Mini head camera. The tasks cover basic pick-and-place, hanoi tower sorting, and Ethernet insertion requiring fine alignment and contact handling.

Harness: Engineer, Runner, and human support

Our workflow uses two roles. The Engineer organizes task goals, stage plans, and result evaluation. The Runner writes and executes code, returning physical feedback for the next iteration. Humans provide calibration, interpretations of the scene, or methodological suggestions when needed.

Agents choose their own implementation approach. They may encode more of the task in a program or continue observing, reasoning, and adjusting during execution. The resulting implementation is itself part of what we study.

Exploration and execution

Exploration measures the operator-reported time to first success and human assistance; execution measures success, elapsed time, and agent involvement across five selected runs per model per task. Development conditions were not matched, and Ethernet success denotes mechanical insertion only.

4. Findings

Online model participation can address local states that a program does not yet capture

Ethernet insertion illustrates the value of online judgment. Astra reads raw images and specifies a movement direction and distance. A controller executes the command, a script checks insertion, and the model observes again if needed. Fable 5.1 explored visual alignment and admittance control, but its recorded implementation did not complete insertion. These observations support further study of hybrid execution; isolating the contribution of online participation requires additional controlled comparisons. Replay decisions and agent involvement ↗

Source: website methods, findings, and recorded outcomes. The diagram summarizes the recorded implementations; it does not isolate the causal effect of model participation.

First success and efficient repeated execution are different objectives

Keeping the model involved reduces the need to encode every situation in advance, but repeated observation, reasoning, and confirmation add time. The advantage in reaching first success varies across tasks, and reaching a working solution sooner does not guarantee lower repeated execution costs. Compare how the two agents complete each task ↗

Calibration, verification, and recovery affect exploration efficiency

Astra is more sensitive to calibration discrepancies and spends more time checking them. That caution can prevent errors, but it can also prolong a task that was already feasible. Verification and recovery require a similar balance: checks should reflect actual task requirements and allow reobservation or a local retry when evidence is insufficient.

Agents can improve local manipulation strategies through failure feedback

Agents can define evaluation criteria before an attempt and modify their approach afterward. Changes extend beyond parameters to poses and action sequences. For example, after vertical disc handling caused interference, an agent adopted a tilted approach, turning physical feedback into a concrete change in the next attempt.

Local improvement does not guarantee effective management of exploration

Agents can spend a long time optimizing one method while rarely comparing alternatives proactively. They may abandon a method after prolonged lack of progress, but timely switching usually requires a human request. Diagnosing failure, improving the current method, and deciding whether that method remains worth pursuing are distinct capabilities.

Humans provide information and influence how agents explore

Human help supplies physical information, interprets observations, and redirects exploration. After a human calibrated the fixed post position, for example, Fable could handle discs placed randomly within the workspace.

Expectations of help also affect behavior. When grasping clearance was insufficient, Fable initially asked a human to move the disc. After being told that no assistance would be provided, it closed the gripper and pushed the disc into a more accessible position. The content and timing of assistance are more informative than a binary label of human involvement.

5. Discussion and Open Questions

Assign responsibilities according to the nature of the decision

High-frequency motion, explicit constraints, and mature local control are well suited to programs. Models are better placed where a decision requires interpreting observations, diagnosing an exception, or changing an approach. As the relevant conditions and methods become clearer, stable decisions can be captured in reusable code.

How should experience be preserved across sessions?

Some of a successful solution resides in code; other parts may reside in the current conversation, experience documents, and implicit operating conditions. A fresh session may need further attempts to recover previous performance, even with earlier records. Conversely, restarting can sometimes improve the speed of a long-running session.

Which information belongs in active context, which should become code or external experience, and which is outdated? We need to evaluate how preserving experience affects completion, recovery, and execution time to establish a reliable handoff process.

How should efficiency and precision be balanced?

More observations, smaller movements, and additional checks usually cost time, without always producing a corresponding benefit. Error tolerance also changes across a task: an offset that is acceptable during transport may determine success during final insertion.

A next step is to investigate whether the system can allocate observation, reasoning, and control effort according to task stage and uncertainty. This requires measuring completion, execution time, and human effort under the same success criteria to establish whether an approach improves real-world working efficiency.

@misc{wang2026agentsrealworld,
  author = {Qiang Wang and Jianxin Bi and Shulong Jiang and Wenbo Huang and Shunyu Yang and Ruiyang Si and Rui Ni and Daquan Zhou},
  title = {{Agents in the Real World: From Exploration to Execution}},
  year = {2026},
  month = {September},
  day = {20},
  url = {https://dagroup-pku.github.io/Agentic-Robot/posts/agents-in-the-real-world/}
}