Code-as-Policies with Astra/Fable in RoboDojo

1. Summary of results

18 tasks · 20 rollouts per task, per method. Frozen policies, with no coding agent involved during evaluation.

Both agents turn task instructions into executable robot policies. Symbolic reasoning is a relative strength; randomized scenes, grasp recovery, and contact-rich manipulation remain difficult.

Video: an earlier development highlight, not a recording from the scored evaluation batch.

2. Evaluation setup

From RoboDojo’s 54 settings, we selected 18 where Astra showed development progress under the nominal ten-iteration coding scheme. This is a selected panel, not an evaluation of all 54 settings.

Develop → freeze → evaluate. Agents write code and iterate through evaluator feedback. The nominal budget is ten development episodes—ten one-episode submissions; larger submissions consume more trials. We freeze one policy per task, then evaluate it on 20 scene layouts. The native task evaluator decides success. Perception, planning, checks, and programmed recovery still run locally.

Benchmark reference: RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies.

3. Main results in detail

• Symbolic tasks: Astra 66.25%, Fable 57.50%. Both reach 80% on tic-tac-toe; numbered-button pressing reaches 100% and 90%.
• Different category profiles: Astra leads on block manipulation, 79% vs 47%. Fable leads on cloth folding, 60% vs 30%, and shape manipulation, 50% vs 25%.
• Randomization hurts both: block stacking drops from 85% → 40% for Astra and 35% → 30% for Fable. Cloth folding drops from 50% → 10% and 65% → 55%.
• Precision contact and tool use: neither policy set succeeds at coin insertion or xylophone playing. Some failures occur during perception or grasp preparation, before final contact.

Example: the tic-tac-toe policy

The audited Astra controller combines eight shared primitive groups into a task-specific policy. Select a stage to see how its skills connect observations to actions.

4. Findings

Code makes explicit reasoning reusable. Agents combine symbolic decisions with perception and motion, then execute without further agent calls. Moving agent reasoning into development removes repeated model waiting from the rollout.

Offline vs purely online execution

Astra in both modes; 70 episodes each across 14 settings, including failures. Online tokens sum input and output across all calls, including cached input (about 86,000 tokens per rollout). These are evaluation costs; offline policy development uses tokens separately.

Online demonstrates broader task completion, with latency in the action loop. The online reference succeeds at least once on 32/54 settings; our frozen Astra policies do so on 14/18 selected settings. The panels and protocols differ. Online observation→model→action cycles allow fresh reasoning, but each decision waits for inference and the API response. This dependence is a structural latency constraint of agent-in-the-loop control; the exact delay depends on the model and serving system. Coverage and trade-offs ↗

Reliable physical interaction remains the bottleneck. Scene variation, grasp retention, and contact preparation can break an otherwise sensible plan. Runtime policy errors indicate checks stopping running code, not compilation failures; catching an error also does not guarantee task success. Our evidence does not isolate whether stricter checks help or hurt.

Explore the RoboDojo overview for context, or inspect the task-level results, recordings, and primitive graphs.

@misc{bi2026codeaspolicies,
  author = {Jianxin Bi and Qiang Wang and Rui Ni and Wenbo Huang and Shunyu Yang and Ruiyang Si and Shulong Jiang and Daquan Zhou},
  title = {{Code-as-Policies with Astra/Fable in RoboDojo}},
  year = {2026},
  month = {September},
  day = {20},
  url = {https://dagroup-pku.github.io/Agentic-Robot/posts/robodojo-offline-policy-zero-shot/}
}