RoboTwin 2.0

Runs on RPent

RoboTwin 2.0 runs on RPent's RoboTwin stack: RPent boots the SAPIEN environment and the LingBot-VLA, and every robo call goes through RPent's toolkit. New robots are added through XPolicyLab (Adding a Robot).

RoboTwin 2.0 is a dual-arm manipulation benchmark in SAPIEN. The robot has two arms with RPent's collision-aware motion planning and the frozen LingBot-VLA.

Overview

Robot	Two arms in SAPIEN (headless, Vulkan)
Action models	LingBot-VLA (lingbot_act)
Agent runtimes	codex (default), claude, openai (Agent Configuration)
Tasks	RoboTwin 2.0's 50 dual-arm tasks, in the demo_randomized evaluation setting
Hardware	Linux, NVIDIA GPU with Vulkan; Python 3.11

Observation and Action

Item	Description
Observation	The head, left_wrist and right_wrist images (240 × 320), their depth and per-pixel world coordinates, and the state of both arms.
Action	The agent's cells call robo; each call is one RPent tool (a planned arm motion, a gripper command or a LingBot-VLA run) that steps the simulator.
Success	RoboTwin's own eval_success predicate. robo.done turns true once it fires, and every further motion is refused.
Task prompt	The task instruction, a task card (the step-0 images and the arms' initial poses) and the robot's knowledge text.

Installation and Resources

RPent at commit 886b3b2, in its own Python 3.11 environment (the same torch build as for LIBERO-PRO), with PyRUA-Lean in the same environment:

uv pip install -e ".[rlinf]" rlinf-robotwin-runtime==0.1.1 rlinf-lingbotvla==0.1.1 imageio-ffmpeg
git clone --branch v0.7.8 https://github.com/NVlabs/curobo.git
uv pip install --no-build-isolation ./curobo         # against that torch: CUDA_HOME (12.8) and TORCH_CUDA_ARCH_LIST set
uv pip install -e /path/to/PyRUA-Lean
robotwin-download-assets --output /path/to/robotwin-assets      # the Hugging Face dataset TianxingChen/RoboTwin2.0, 16 GB
export ROBOTWIN_ASSETS_PATH=/path/to/robotwin-assets

Rendering is headless SAPIEN on Vulkan: the NVIDIA ICD must be found; set VK_ICD_FILENAMES when it is not under /usr/share/vulkan/icd.d.

VLA Configuration

LingBot-VLA RLinf/LingBot-VLA-RoboTwin-EEF-ckpt1500 at revision e727b46cd220b66981ea4d2fd9ba84adc189e2cc, including qwen_base/:

hf download RLinf/LingBot-VLA-RoboTwin-EEF-ckpt1500 --revision e727b46cd220b66981ea4d2fd9ba84adc189e2cc \
    --local-dir /path/to/lingbot
export LINGBOT_MODEL_PATH=/path/to/lingbot

Task Selection

Option	Meaning
--task	A RoboTwin task name, e.g. beat_block_hammer.
--suite	The task config (default demo_randomized, the evaluation setting).
--seed	The exact scene seed, e.g. 100000; RPent lists verified seeds in robots/robotwin/eval/demo_randomized.json. Always pass it: pyrualean play sends 0 when it is left out.

Run a Task

export RPENT_ROOT=/path/to/rpent
pyrualean play --robot robotwin --arm cells --task beat_block_hammer --suite demo_randomized --seed 100000 \
    --model gpt-6-astra --max-turns 40 --budget-s 7200 --out runs/robotwin-demo --cuda-device 0
pyrualean summarize runs/robotwin-demo

An episode boots private env and VLA servers (~7 GB + ~9 GB of GPU memory); --vla-endpoint attaches it to a running LingBot-VLA server instead.

View Results

The run directory keeps the episode video (episode.mp4), every cell the agent ran (workspace/) and what each printed (the runtime's event file, e.g. codex_events.jsonl), the ledger of primitive calls (calls.jsonl), the token usage and RoboTwin's verdict (result.json), and RPent's own records (states.json, per-step images and world maps).

Results

GPT-6 Astra, 50 tasks at five seeds (250 task instances), at most 40 LLM calls and two hours per episode. Against RPent's tool-calling agent with the same planner and primitives; LLM calls, input tokens and cost per solved episode on the instances both solved:

	Success (%)	LLM calls	Input tokens	Cost
Tool calling (RPent)	60.0	12.2	415k	$1.00
PyRUA-Lean	69.2	8.9	245k	$0.66

What Runs Where

• Env server (RPent) owns the RoboTwin task environment and its SAPIEN rendering.
• LingBot-VLA server (RPent) holds the VLA.
• PyRUA-Lean boots them for the episode, builds RPent's toolkit and serves the agent's cells against robo.
• Agent runtime (Codex CLI, Claude Code or the openai loop) talks to the model.

Primitives the Agent Can Call

Motion:

• move_to(arm, xyz, quat=None, gripper=None): plan and execute a collision-aware motion of one arm to a world pose.
• rotate_wrist(arm, delta_yaw_deg): rotate one end-effector about the world z axis, keeping its position.
• set_gripper(arm, val, steps), release(arm): drive one gripper; open it.
• lingbot_act(chunks): let the frozen LingBot-VLA drive both arms for chunks × 50 native actions.

Perception and state:

• sample_world_xyz(view, pixels), query_world_map(view, bbox): world points under pixels, statistics of the points inside a pixel box.
• state(), image(camera), depth(camera), world_map(camera), render(), show(camera), camera_meta(camera), artifact(name), done.

API reference: what the agent reads, generated from pyrualean.robotwin when these docs are built

class RobotwinRobot
    A RoboTwin dual-arm robot (aloha-agilex: two 6-DoF arms with parallel grippers) at a table.

    World frame: x runs across the table (the robot's LEFT arm is at negative
    x and appears on the LEFT of the head image), y points away from the robot
    towards the far edge of the table, z is up; metres and radians.
    Orientations are unit quaternions ordered [w, x, y, z].  The arms are
    ``"left"`` and ``"right"`` (``robo.LEFT`` / ``robo.RIGHT``); every motion
    call names one arm and leaves the other where it is.

    Two frames per arm: ``move_to`` targets the EEF (wrist link); the TCP
    (gripper centre, where an object is held) lies ``EEF_TO_TCP_M`` = 0.12 m
    beyond the EEF along the gripper's approach axis (``ArmState.approach``),
    so to bring the gripper centre onto a point p command
    ``p - 0.12 * approach``.  Never send an object's surface point as an EEF
    target.  Both frames are reported by ``state()``.

    Gripper commands are floats in [0, 1]: ``robo.OPEN`` (1.0) is fully open,
    ``robo.CLOSE`` (0.0) fully closed; ``state().left.gripper`` reports the
    commanded value, not a contact measurement.

    Every native action (one ``move_to`` substep, one gripper step, one VLA
    action) advances ``take_action_cnt`` towards ``step_lim``.  Motion calls
    block until the primitive finishes and return a typed result; nothing is
    queued.

    Perception is camera based: a fixed ``head`` camera (whole table, 320x240)
    and one wrist camera per arm (``left_wrist``, ``right_wrist``); every
    recorded step also carries per-pixel world coordinates for each view
    (``sample_world_xyz``, ``query_world_map``, ``world_map``).  Object poses
    are never given.

    The frozen LingBot-VLA (``lingbot_act``) is a learned dual-arm policy for
    this benchmark.  It always receives the benchmark's own task instruction
    (``robo.task``), never a prompt of yours.

Constants:
    robo.ARMS = ('left', 'right')
    robo.LEFT = 'left'
    robo.RIGHT = 'right'
    robo.OPEN = 1.0
    robo.CLOSE = 0.0
    robo.EEF_TO_TCP_M = 0.12

Methods and attributes of `robo`:

robo.task -> str  (read-only attribute)
    The task instruction, e.g. ``"Grab the hammer with the left arm and hit the block"``.

    It is authoritative: it names the arm when one is required and the VLA
    acts on exactly this text.

robo.done -> bool  (read-only attribute)
    True once the benchmark's success predicate (``eval_success``) has fired (sticky).

robo.state(step: int = -1) -> State
    Read the recorded state of both arms and the episode status.

    Args:
        step: recorded step to read; ``-1`` is the latest, ``0`` the
            initial scene.  A new step is recorded after every motion call
            and after ``render``.
    Returns:
        State with ``left`` / ``right`` arm poses and gripper values,
        ``take_action_cnt`` / ``step_lim`` and the episode flags.

robo.image(camera: str = 'head', *, resolution: str = 'high', step: int = -1) -> Any
    Load a recorded camera image as a ``numpy`` uint8 array (240, 320, 3), RGB.

    Args:
        camera: ``"head"`` (fixed view of the whole table: identify objects,
            destinations and the global layout), ``"left_wrist"`` or
            ``"right_wrist"`` (on the gripper: close-range geometry of the
            object that arm is working on).
        resolution: there is one resolution per camera, ``"high"`` (320x240).
            Pixel coordinates are (row, col) with row 0 at the top and col 0
            at the left; pass the same camera and step to
            ``sample_world_xyz`` / ``query_world_map``.
        step: recorded step, ``-1`` = latest (call ``render()`` first to
            get a fresh picture after time has passed without motion).

robo.world_map(camera: str = 'head', *, resolution: str = 'high', step: int = -1) -> Any
    Load the per-pixel world coordinates of a recorded image.

    Returns a ``numpy`` float32 array (240, 320, 3): ``world_map[row, col]``
    is the [x, y, z] of the surface seen at that pixel, NaN where the
    camera saw no depth.  This is what ``sample_world_xyz`` and
    ``query_world_map`` read; load it to reason about many pixels at once
    (e.g. ``np.nanmedian(world_map[..., 2])`` is the table height).

robo.show(camera: str = 'head') -> None
    Attach the current image of ``camera`` to the feedback of this turn.

    Images are not returned automatically; call ``show("head")``,
    ``show("left_wrist")`` or ``show("right_wrist")`` when *you* (the
    author) need to look at the scene before writing the next turn.  Costs
    tokens: look only when a decision depends on it.

robo.artifact(name: str, *, step: int = -1) -> bytes
    Return the raw bytes of a recorded artifact (advanced).

    Names are ``<camera>_rgb.png``, ``<camera>_depth.npy``,
    ``<camera>_world_xyz.npy`` and ``<camera>_camera_meta.json`` for the
    three cameras.

robo.depth(camera: str = 'head', *, step: int = -1) -> Any
    Load the metric depth of a recorded image: float32 (240, 320) metres, NaN if invalid.

robo.camera_meta(camera: str = 'head', *, step: int = -1) -> dict[str, Any]
    Return the camera calibration of a recorded step as a dict.

    Keys: ``intrinsic_K`` (3x3), ``extrinsic_cv`` (3x4), ``cam2world_gl``
    (4x4, OpenGL convention: the camera looks down its -z axis), ``width``,
    ``height``.  The wrist cameras move with their arm.

robo.render() -> State
    Record a fresh observation (new step with images and world maps) without moving.

    Use it before ``image`` / ``sample_world_xyz`` when the scene may have
    changed since the last recorded step (an object settling after a
    release, for instance).  Costs no native action.

robo.sample_world_xyz(
    view: str,
    pixels: Sequence[Sequence[int]],
    *,
    step: int = -1,
    neighborhood: int = 1,
) -> WorldSamples
    Return the world [x, y, z] under image pixels of a recorded view.

    Args:
        view: ``"head"``, ``"left_wrist"`` or ``"right_wrist"``; it is also
            the pixel coordinate space, so use the view whose image the
            pixels were read from (same ``step``).
        pixels: 1 to 256 ``[row, col]`` pairs (a single pair is accepted too).
        step: recorded step, ``-1`` = latest.
        neighborhood: half-size of the square window (0..32) whose finite
            points are median-filtered; 1 = a 3x3 window.
    Returns:
        WorldSamples; each ``xyz`` is a visible SURFACE point (the top of an
        object, not its centre): sample several interior pixels and take
        the median; pixels on edges, shadows or the gap to the table hit
        the background.
    Raises:
        ToolError: pixel out of bounds (images are 240 rows x 320 cols),
        no finite depth in the window, or no world map for that step.

robo.query_world_map(
    view: str,
    bbox: Sequence[int],
    *,
    step: int = -1,
    max_points: int = 256,
) -> WorldRegion
    Return statistics of the world points inside a pixel box of a recorded view.

    Args:
        view: ``"head"``, ``"left_wrist"`` or ``"right_wrist"`` (the box's
            pixel space; same step as the image you read it from).
        bbox: half-open ``[row_start, col_start, row_end, col_end]``.
        step: recorded step, ``-1`` = latest.
        max_points: how many ``(pixel, xyz)`` samples to return (1..4096).
    Returns:
        WorldRegion; ``xyz_median`` is the robust surface height / position
        of what fills the box, ``xyz_min`` / ``xyz_max`` its extent.
    Raises:
        ToolError: box outside the image or no finite depth inside it.

robo.lingbot_act(
    chunks: int = 4,
    *,
    use_length: int = 50,
    prompt: str | None = None,
) -> VlaRun
    Let the frozen LingBot-VLA drive both arms for ``chunks`` x 50 native actions.

    The policy runs on the benchmark's own task instruction (``robo.task``)
    and the three current camera images; it stops early when the task
    succeeds or the budget runs out.  It is the tool for grasps and
    re-grasps, bimanual coordination, insertion, hanging, tool use and
    other contact-rich motion; script free-space transport, staging and
    release yourself with the primitives after verifying the state.

    Args:
        chunks: action chunks to execute (>= 1).  One chunk near contact,
            near success or for a small correction; two for ordinary
            stable progress; three only for a continuity-sensitive phase
            that is already moving correctly.
        use_length: actions per chunk; the runtime fixes it at 50.
        prompt: recorded with the call but NOT sent to the policy (the
            runtime always uses the task instruction); leave it None.
    Returns:
        VlaRun; check ``terminated`` / ``robo.done`` and the images
        afterwards - a completed run is not a solved task.

robo.move_to(
    arm: str,
    xyz: Sequence[float],
    *,
    quat: Sequence[float] | None = None,
    gripper: float | None = None,
    substeps: int = 25,
) -> Move
    Plan and execute a collision-aware arm motion to a world-frame EEF pose.

    The motion planner computes a joint path from the current pose to the
    target; the path is executed as ``substeps`` native actions.  If no
    path exists (unreachable, in collision, through the other arm or the
    table) nothing moves and ``Move.planned`` is False.

    Args:
        arm: ``"left"`` or ``"right"``.
        xyz: target EEF position [x, y, z] in metres (NOT the gripper centre:
            subtract 0.12 m along ``ArmState.approach``).
        quat: target orientation [w, x, y, z]; None keeps the current one.
        gripper: gripper value in [0, 1] to hold during the motion; None
            keeps the current command (so a held object stays held).
        substeps: waypoints executed (1 = jump to the final pose in one
            action, 0 = every planner waypoint).  Near the table, a rim or
            the other arm use small motions (z changes of 5-10 mm) with at
            most 8 substeps and re-observe between them.
    Returns:
        Move; always check ``planned`` and ``reached`` and compare
        ``final_eef_pos`` with the target: a returned call is not proof of
        arrival.

robo.rotate_wrist(
    arm: str,
    delta_yaw_deg: float,
    *,
    gripper: float | None = None,
    substeps: int = 25,
) -> Move
    Rotate one EEF about the world z axis by a relative angle, keeping its position.

    Because the TCP sits 0.12 m from the EEF, the gripper (and a held
    object) sweeps an arc: rotate only with clearance around the gripper,
    at a safe height, in small increments, after a verified hold.

    Args:
        arm: ``"left"`` or ``"right"``.
        delta_yaw_deg: relative yaw in degrees (``state().arm(arm).eef_yaw``
            reports the current yaw in radians).
        gripper: value to hold during the motion; None keeps the current.
        substeps: waypoints executed (see ``move_to``).
    Returns:
        Move (``target_xyz`` is the position it was told to keep).

robo.set_gripper(arm: str, val: float, steps: int = 10) -> Gripper
    Drive one gripper linearly to ``val`` over ``steps`` native actions.

    ``set_gripper("left", robo.CLOSE)`` closes the left gripper; verify a
    grasp afterwards from the wrist image and from the object moving with
    the TCP - the reported value is a command, not a contact measurement.

robo.release(arm: str, val: float = 1.0, steps: int = 10) -> Gripper
    Open one gripper to ``val`` (default fully open) over ``steps`` native actions.

    Release only when the object is supported at its destination; then
    retreat upwards and check that it stayed put.  Returns
    ``terminated=True`` when letting go satisfied the task.

Result types (frozen dataclasses; access fields as attributes):

class State(
    step: int,
    task: str,
    left: ArmState,
    right: ArmState,
    eval_success: bool,
    take_action_cnt: int,
    step_lim: int,
    terminated: bool,
    truncated: bool,
)
    Snapshot of both arms and of the episode at one recorded step.

    step: index of the recorded step (0 = initial scene).  A new step is
        recorded after every motion call and after ``render``.
    task: the task instruction given by the benchmark (authoritative; the VLA
        acts on exactly this text).
    left / right: the two arms (``state.arm("left")`` picks one by name).
    eval_success: the benchmark's success predicate has fired (sticky).  This
        is the only definition of success; it equals ``terminated``.
    take_action_cnt: native actions executed so far.  Every ``move_to``
        substep, every gripper step and every VLA action is one native action.
    step_lim: native-action budget of the episode.
    terminated: same as ``eval_success``.
    truncated: the native-action budget is exhausted (``take_action_cnt >=
        step_lim``); no further motion is possible.
    .actions_left -> int: Native actions still available before ``truncated`` fires.

class ArmState(
    eef_pos: tuple[float, float, float],
    eef_quat: tuple[float, float, float, float],
    tcp_pos: tuple[float, float, float],
    tcp_quat: tuple[float, float, float, float],
    gripper: float,
)
    One arm at a recorded step.

    eef_pos: end-effector position [x, y, z] in the world frame, metres.  The
        EEF is the wrist link that ``move_to`` targets.
    eef_quat: EEF orientation as a unit quaternion [w, x, y, z].
    tcp_pos: tool centre point (the gripper centre, where an object is held):
        0.12 m beyond the EEF along the gripper's approach axis.
    tcp_quat: TCP orientation [w, x, y, z] (same as the EEF).
    gripper: normalised gripper command, 0.0 = closed ... 1.0 = open.  It is
        the position the fingers were told to hold, not a contact sensor.
    .approach -> tuple[float, float, float]: Unit vector from the EEF to the TCP: the direction the gripper points (world frame).
    .eef_yaw -> float: Rotation of the EEF about the world z axis, radians (what ``rotate_wrist`` changes).

class Move(
    arm: str,
    target_xyz: tuple[float, float, float],
    final_eef_pos: tuple[float, float, float] | None,
    final_dist_m: float,
    planned: bool,
    reached: bool,
    waypoints: int,
    actions_used: int,
    stop_reason: str,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``move_to`` / ``rotate_wrist``.

    planned: the motion planner found a collision-free joint path.  False
        means NOTHING moved (target unreachable, in collision, or too close to
        the other arm / the table): change the target, do not repeat it.
    final_eef_pos: where the EEF actually stopped (metres), None when nothing
        moved.
    final_dist_m: distance between the requested xyz and ``final_eef_pos``
        (``rotate_wrist``: distance from the xyz it was told to hold).
    reached: ``planned`` and ``final_dist_m <= 0.01``.  A planned motion can
        still stop short when the episode ended or the budget ran out.
    waypoints: planner waypoints that were executed.
    actions_used: native actions consumed (one per waypoint).
    stop_reason: ``completed`` | ``native_success`` | ``budget_exhausted`` |
        ``plan_failed`` | ``runtime_failure``.
    terminated / truncated: episode flags after the motion.

class Gripper(
    arm: str,
    target: float,
    final: float,
    actions_used: int,
    stop_reason: str,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``set_gripper`` / ``release``.

    target: the commanded gripper value; final: the value reported after the
        motion (a commanded position, not proof of a hold: verify a grasp from
        the images and from the object moving with the TCP).
    actions_used: native actions consumed (one per interpolation step).

class VlaRun(
    chunks_requested: int,
    actions_requested: int,
    actions_used: int,
    stop_reason: str,
    instruction: str,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``lingbot_act``.

    chunks_requested / actions_requested: what was asked (50 actions per chunk).
    actions_used: native actions actually executed; fewer than requested when
        the task succeeded (``native_success``) or the budget ran out.
    stop_reason: ``completed`` | ``native_success`` | ``budget_exhausted`` |
        ``runtime_failure``.
    instruction: the task text the policy received (always the benchmark's
        own instruction, i.e. ``robo.task``).
    terminated / truncated: episode flags after the run.

class WorldSamples(
    samples: tuple[WorldSample, ...],
    view: str,
    step: int,
    image_shape: tuple[int, int],
)
    Outcome of ``sample_world_xyz``: one ``WorldSample`` per requested pixel, in order.
    .xyz -> tuple[tuple[float, float, float], ...]: The sampled coordinates only, in request order.

class WorldSample(
    pixel: tuple[int, int],
    xyz: tuple[float, float, float],
    valid_points: int,
)
    World coordinate of one queried pixel (``sample_world_xyz``).

    xyz: median [x, y, z] (metres) of the finite world points in the pixel's
        neighbourhood; valid_points: how many neighbourhood pixels had depth.

class WorldRegion(
    xyz_min: tuple[float, float, float],
    xyz_max: tuple[float, float, float],
    xyz_median: tuple[float, float, float],
    valid_points: int,
    points: tuple[tuple[tuple[int, int], tuple[float, float, float]], ...],
    view: str,
    step: int,
)
    Outcome of ``query_world_map``: statistics of the world points inside a pixel box.

    xyz_min / xyz_max / xyz_median: per-axis extremes and median of the
        finite points (metres); valid_points: how many pixels had depth.
    points: up to ``max_points`` ``(pixel, xyz)`` samples spread over the box
        in row-major order, for your own geometry.

Exceptions:
    ToolError(RuntimeError): a call was rejected or a service failed; catch it if you have a fallback.
    ValueError: an argument was invalid (e.g. a planar move over 0.30 m) - nothing was executed.
    EpisodeFinished(BaseException): the episode is over (reason: terminated | truncated | timeout | cancelled); return.