LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data
A coding agent does the teleoperator's job. One calibration episode in, real-ready robot data out.
Simulation in, consistent multi-view video out
Left: the simulated rollout. Right: the farm episode rendered from it. Head camera on top, wrist cameras below; the views agree with each other, and every frame keeps the exact action behind it. Fold towel plays at 2×.
Video model alone vs. inside DataFarm
The same simulated episode, the same video model. Called directly, it gets the look but not the episode: extra cups, a misplaced arm, views that disagree. Inside DataFarm, every view stays consistent with the others and with the simulation.
What DataFarm adds to the video model
Structure from the simulator, appearance from the real cameras. Edge maps of the untextured simulator video fix arm pose, object positions, and timing; real reference frames and a picture of the robot supply only the look.
All cameras in one generation. Head and wrist views are tiled on one canvas and generated as one video, so they share lighting and materials and agree on which objects exist and what each gripper holds.
References and prompts that add no content, and long episodes without drift: references match the simulated task state, prompts describe only the current segment, and long episodes are generated as independent chunks and joined where neighbouring chunks differ least.
| Renderer | Motion ↑ | Structure ↑ | Jumps ↓ |
|---|---|---|---|
| Direct call, one canvas | 0.07 | 0.16 | 11.2 |
| Direct call, each camera | 0.04 | 0.10 | 8.8 |
| Edge control only | 0.50 | 0.54 | 30.2 |
| DataFarm, each camera† | 0.55 | 0.64 | 40.6 |
| DataFarm | 0.74 | 0.85 | 2.9 |
Correlation of each view with the simulator video, in frame-difference energy (motion) and image gradients (structure); jumps per 1,000 frames. Means over stack cups, weigh apple, and pull drawer. †Stack cups only; DataFarm on that episode: 0.75, 0.87, 4.2.
Robot data is still collected one episode at a time
- Each episode needs a robot, an operator, and a reset.
- Every new task, layout, or scene starts the collection over.
- The cost grows with every episode you record.
A coding agent takes the teleoperator's place
It does not drive the robot: it compiles the task into a simulation and a render contract, and the farm produces every further episode.
One calibration episode: a short recording of the robot in the target scene, plus a request such as "100 episodes of this task". None of its actions are reused.
The agent fits cameras and scene, qualifies the simulator, and writes an expert against a judge it cannot see; the simulator fixes actions, states, and contacts.
A video model conditioned on real frames renders every rollout as photographic video, all cameras in one generation.
Machine checks and an independent reviewer compare every render with its simulated state; only episodes that agree enter the dataset.
From simulation to photograph
Same rollout, real look
Drag across the video. Left of the handle: the simulator rollout; right: the farm episode rendered from it, frame by frame.
Texture the simulator cannot draw
The simulator models the geometry and contacts the physics needs; the look of a lit display or a woven towel comes from the calibration episode, set once per task by a render contract costing $4.1–6.2.
Simulator
Real camera
DataFarm
Simulator
Real camera
DataFarmReal frames come from a recording at the same task phase, not the same frame.
One calibration episode, one hundred farm episodes
Nine stack-cups farm episodes, each a different simulated rollout with its own layout. Once the task is compiled, every further episode needs no robot and no operator.
New appearance, same motion
A new tabletop or light only changes the render request; the simulated motion stays the same. Teleoperation would have to restage the scene for each one.
Inset: the simulator video of the same rollout.
Nothing enters the dataset on the agent's word
A generated video can look right and still contradict its labels. Machine checks and an independent reviewer compare every render with its simulated state, and reject failures like these.
Phantom arm
An arm the simulator does not contain.
simulator
render: rejectedFuture state drawn early
The block is in the drawer before the simulator puts it there.
simulator
render: rejectedInvented cloth
A wrist view adds towel where the simulation is empty.
simulator
render: rejected
Closer to the real cameras. Kernel distance of head-view frames to real camera frames (KID ×103): 0.83–0.88 for farm episodes, against 1.93 for the simulator and 2.50 with domain randomization (Sim + DR). The farm is closer than the simulator on every task.
The simulated motion is kept. Farm renders deviate from the simulator's optical flow by 4.4 pixels and agree on 65% of moving pixels, close to Sim + DR (3.4, 73%), which has exactly the simulated motion; paired with the simulator video half a second later they reach 8.8 pixels and 41%.
Does farm data work on a real robot?
Real-ready: trained only on farm data, run directly on the robot
π0.5 policies trained on 100 episodes from each source, 20 trials per task and condition, on a dual-arm PiPER robot. Neither training set contains the test tablecloth or the light.
On par in the original scene, far ahead under unseen appearance. Farm-trained policies match or exceed teleoperated data on four of five tasks in the original scene. Under the unseen tablecloth and light, they lead on every task, by 11, 6, 12, 7, and 5 of 20 trials.
Rendering with real images is what transfers. Sim + DR shares the grounding, the expert, and the actions with DataFarm and differs only in how the frames are produced. The farm varies surfaces and lighting around fixed, exact trajectories, so the policy cannot rely on how the table looks.
Pay once per task, then only per episode
The farm pays once to adapt a task, with no person in the loop, then only for simulating, rendering, and reviewing each episode; a new appearance is a text change. The farm's figures below count only published or measured inputs; the teleoperation figures are labour in North America and a quote from China.
| Task | One-time | Contract | Farm / ep. | Avg. / ep. N=100 | Avg. / ep. N=1,000 | Per hour 1 h / task |
|---|---|---|---|---|---|---|
| Stack cups | 18.22 | 6.11 | 0.93 | 1.17 | 0.96 | 101 |
| Weigh apple | 10.33 | 4.72 | 0.89 | 1.04 | 0.91 | 117 |
| Beat drum | 19.25 | 4.11 | 0.26 | 0.49 | 0.28 | 106 |
| Pull drawer | 19.40 | 4.49 | 0.52 | 0.76 | 0.54 | 79 |
| Fold towel | 37.38 | 6.23 | 1.41 | 1.84 | 1.45 | 132 |
| Mean | 20.92 | 5.13 | 0.80 | 1.06 | 0.83 | 107 |
| Teleop., N. America* | – | – | 2.86 | 2.86 | 2.86 | 290 |
| Teleop., China, quote | – | – | 1.16 | 1.16 | 1.16 | 117 |
USD. One-time: agent cost from the calibration episode to five admitted simulated episodes, at standard API pricing, with Claude Opus 5.5 in SoL-Pi, and in Claude Code on fold towel. Contract: the render contract. Farm / ep.: rollouts and audit, H3 rendering on rented GPUs, re-rendering of flagged chunks, and independent review per admitted episode. Avg. / ep.: one-time and contract costs spread over N episodes plus the per-episode cost; it falls towards the per-episode cost. Per hour, 1 h / task: the cost of collecting one hour of synchronized three-view data for a task, one-time and contract costs included. Teleop., N. America: *estimated with Claude Opus 5.5, the labour of teleoperated collection (operator, supervision and review, post-processing; no robot, venue, or setup), $190–520 per usable hour across our assumptions. Teleop., China: a full-service quote for custom data, about RMB 800 per hour. Both per episode at the farm's mean length.
Certification is kept apart from production
Because agents write both the expert and its checks, and the video model may alter the scene, every stage is checked by a component other than its producer.
- Grounding and simulator qualification. The agent fits the cameras and metric scale, estimates object geometry and articulation, and labels each quantity as measured, fitted, assumed, or unknown; short probes check that the simulator can reproduce the skills the task requires.
- Task contract and expert. The expert is developed against a judge it cannot inspect and admitted only on held-out initial states; every admitted trajectory is one continuous execution.
- Rendering with real images. The renderer may alter appearance but not geometry, objects, or timing; all cameras are generated together from the simulator's structure.
- Verification and admission. Every rendered episode is compared with its simulated state by machine checks and an independent reviewer; all attempts and failures are kept.
What the farm delivers. (a) The same frame of a simulator rollout and of the farm episode for each task. (b) From a request and one calibration episode, the agent builds the task in simulation, and the video model renders each rollout in the original scene or a scene described by text. (c) Real-robot successes. (d) Mean costs over the five tasks.
The DataFarm pipeline, followed on one drawer episode. The capability layer is shared by all tasks; everything above it is written for this task.
Following the simulation in every view. Red boxes mark views that contradict the simulator or the other views of the same frame.
Abstract
Teleoperation remains the default way to collect robot demonstrations, but every episode costs robot and operator time, and every new task starts from scratch. We ask whether a coding agent can take over the teleoperator's job. In our DataFarm, the agent receives a data request and one calibration episode of the robot in the target scene and rebuilds the task in simulation: it writes the scene, an expert controller, and a success judge. The simulator supplies exact actions, and a video model conditioned on real images renders each rollout as photographic video, generating all camera views together consistently. The output is real-ready data: synchronized camera views labelled with the robot's actions, on which a policy can be trained directly and deployed on the real robot. Independent judges certify the expert and every rendered episode before admission. On five real-robot tasks, policies trained only on farm data match policies trained on as many teleoperated episodes (93 vs. 84 successes in 100 trials), and under an unseen tablecloth and coloured light they keep 87 successes where the teleoperation-trained policies fall to 46. Adapting a task is fully automatic and costs $21 of API usage on average, paid once per task, plus $5 for a render contract; every further episode costs $0.80 on average for simulation, rendering, and review, with no robot or operator. One hour of three-view data for a task costs $107 on average, these costs included, slightly below a quote of $117 for teleoperation in China and below $290 in labour alone in North America.
BibTeX
@article{zhang2026datafarm,
title={PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data},
author={Zhang, Kewei and Ma, Juncheng and Lv, Guodong and Wang, Duomin and Ruan, Feixiang and Si, Ruiyang and Zhang, Chuanrui and Chen, Yuxuan and Chen, Hao and Liu, Yichen and Zhang, Yintianrun and Deng, Yufan and Xie, Enze and Zhou, Daquan},
journal={arXiv preprint},
year={2026}
}One calibration episode in. Real-ready robot data out.