RoboCasa365

Runs on RPent

RoboCasa365 runs on RPent's RoboCasa stack: RPent boots the kitchen environment and the RLDX-1 VLA, and every robo call goes through RPent's toolkit. New robots are added through XPolicyLab (Adding a Robot).

RoboCasa365 is a household benchmark of kitchen scenes. The robot is a mobile manipulator: a mobile base and an arm, with navigation, scripted arm motions and the frozen RLDX-1 VLA.

Overview

Robot	PandaOmron mobile manipulator in RoboCasa's kitchens (MuJoCo / robosuite)
Action models	RLDX-1 (rldx_skill, rldx_arm)
Agent runtimes	codex (default), claude, openai (Agent Configuration)
Tasks	The Target50 set: 18 atomic and 32 composite tasks
Hardware	Linux, NVIDIA GPU, EGL; Python 3.10 (RLDX-1 requires it)

Observation and Action

Item	Description
Observation	The agentview, navview and wrist images (256 × 256), their depth and per-pixel world coordinates, and the state of the base, the arm and the episode.
Action	The agent's cells call robo; each call is one RPent tool (navigation, a base or arm motion, a gripper command, an RLDX-1 skill) that steps the simulator.
Success	RoboCasa's own task predicate. robo.done turns true once it fires, and every further motion is refused; success_criteria() returns the predicate's source, as RPent's agent can read it too.
Task prompt	The task instruction, a task card (the step-0 images and the robot's initial pose) and the robot's knowledge text.

Installation and Resources

RPent at commit 886b3b2, in its own environment with Python 3.10 (RLDX-1 requires it) and the same torch build as for LIBERO-PRO, with PyRUA-Lean in the same environment. RPent's [robocasa] extra follows three moving branches; install .[rlinf] and those three sources at the commits the paper used instead, with the Target50 constraints (RPent's pyproject allows pre-releases):

uv pip install -e ".[rlinf]" \
    "rpent-robocasa365 @ git+https://github.com/RLinf/robocasa.git@2692d8fc5fd86708a1b2028dcbce892ec418a9e7" \
    "rlinf-rldx @ git+https://github.com/RLinf/RLDX-1.git@ebcfd13df5177e4b3e574bdf5a3b427c7c4a1e8a" \
    "robosuite @ git+https://github.com/RLinf/robosuite.git@97cfbde4b68d8ec43dad20cf4747297866a6ca2e" \
    --constraint robots/robocasa/eval/target50-constraints.txt --prerelease=if-necessary-or-explicit
uv pip install -e /path/to/PyRUA-Lean
robocasa-download-assets --assets-path /path/to/robocasa-assets --no-macros      # 23 GB

The environment (both servers pin their GPU from --cuda-device, so leave CUDA_VISIBLE_DEVICES unset):

export ROBOCASA_ASSETS_PATH=/path/to/robocasa-assets
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 MUJOCO_GL=egl PYOPENGL_PLATFORM=egl ROBOT_PLATFORM=ROBOCASA NO_ALBUMENTATIONS_UPDATE=1
export RLDX_MAX_CHUNKS=40 RLDX_SETTLE_PATIENCE=999 RLDX_ACTION_STEPS_PER_CHUNK=8     # the Target50 execution values
unset RLDX_RESET_SEED RLDX_ALLOW_RESET CUDA_VISIBLE_DEVICES

Reproducible scenes

Unpatched, RoboCasa can build a different scene for the same task and seed from run to run. Patch the installed file once (idempotent):

sed -i 's/list(set(valid_geoms))/list(dict.fromkeys(valid_geoms))/' \
    "$(python -c 'import importlib.util as u, pathlib as p; print(p.Path(u.find_spec("robocasa").origin).parent / "models/fixtures/counter.py")')"

PyRUA-Lean also starts the env server with PYTHONHASHSEED=0.

VLA Configuration

RLDX-1 at a fixed revision, and the support files (no weights) of its backbone in the Hugging Face cache, at a fixed revision and once more at main (RLDX resolves the backbone at main; without refs/main in the cache the VLA server spends its whole boot budget retrying the Hub):

hf download RLWRLD/RLDX-1-FT-RC365 --revision 587e9ecdcc5e7184fcc17f58713908edff5af041 --local-dir /path/to/rldx
export RLDX_MODEL_PATH=/path/to/rldx
hf download RLWRLD/RLDX-1-VLM --revision 4b9f870d1287e0d38d7eb1445e6d8c60afe66dd7 \
    --include '*.json' '*.txt' '*.jinja' '*.md' '*.png' '.gitattributes'
hf download RLWRLD/RLDX-1-VLM --include '*.json' '*.txt' '*.jinja' '*.md' '*.png' '.gitattributes'

Task Selection

Option	Meaning
--task	A RoboCasa365 task name, e.g. OpenDrawer.
--suite	The data split (default target, the Target50 evaluation setting).
--seed	The scene seed: 1-10 for atomic tasks, 1-5 for composite ones. Always pass it: pyrualean play sends 0 when it is left out.

Run a Task

export RPENT_ROOT=/path/to/rpent
pyrualean play --robot robocasa --arm cells --task OpenDrawer --suite target --seed 1 \
    --model gpt-6-astra --max-turns 40 --budget-s 7200 --out runs/robocasa-demo --cuda-device 0
pyrualean summarize runs/robocasa-demo

An episode boots private env and VLA servers (~3 GB + ~15-17 GB of GPU memory); --vla-endpoint attaches it to a running RLDX-1 server instead.

View Results

The run directory keeps the episode video (episode.mp4), every cell the agent ran (workspace/) and what each printed (the runtime's event file, e.g. codex_events.jsonl), the ledger of primitive calls (calls.jsonl), the token usage and RoboCasa's verdict (result.json), and RPent's own records (states.json, per-step images and world maps).

Results

GPT-6 Astra, five seeds per task; at most 40 LLM calls on atomic tasks and 100 on composite ones, and two hours per episode. Against RPent's tool-calling agent with the same planner and primitives; LLM calls, input tokens and cost per solved episode on the instances both solved:

	Success (%)	LLM calls	Input tokens	Cost
Atomic (90 task instances), tool calling (RPent)	78.9	10.4	261k	$0.62
Atomic, PyRUA-Lean	86.7	5.6	171k	$0.54
Composite (160 task instances), tool calling (RPent)	34.4	28.4	1.22M	$2.01
Composite, PyRUA-Lean	39.4	14.9	603k	$1.25

What Runs Where

• Env server (RPent) owns the RoboCasa kitchen environment and its EGL rendering.
• RLDX-1 server (RPent) holds the VLA.
• PyRUA-Lean boots them for the episode, builds RPent's toolkit and serves the agent's cells against robo.
• Agent runtime (Codex CLI, Claude Code or the openai loop) talks to the model.

Primitives the Agent Can Call

Navigation and motion:

• navigate_to(xy): drive the base towards a world (x, y) and stop facing it; move_base(forward, lateral, turn): raw base velocities.
• move_to(xyz), move_delta(dxyz): servo the gripper to a world position or by a displacement.
• rotate_pitch(target_pitch): tilt the wrist.
• set_gripper(gripper, steps), release(), scripted_grasp(xyz): drive the fingers; let go; a coarse scripted grasp (open, hover, descend, close, lift).
• rldx_skill(): let the frozen RLDX-1 VLA drive the whole body on the task instruction; rldx_arm(): the arm, with the base clamped to small motions.

Perception and state:

• back_project_batch(pixels), query_world_map(z_min, z_max): world points under pixels, what lies inside a height band.
• floor_overlay(): the navigation view with the floor painted.
• state(), image(camera), depth(camera), world_map(camera), show(camera), camera_meta(camera), success_criteria(), artifact(name), done.

API reference: what the agent reads, generated from pyrualean.robocasa when these docs are built

class RobocasaRobot
    A mobile-base PandaOmron (Franka arm with a two-finger gripper on an
    omnidirectional base) in a RoboCasa365 kitchen.

    World frame: x/y span the kitchen (coordinates run over several metres),
    z is up; metres and radians; quaternions are ``[x, y, z, w]``.
    ``state().eef_pos`` is the gripper position in this frame,
    ``state().base_pos`` the base position and ``state().base_yaw`` its
    heading.  Counters are at about z = 0.9 m; read every height from the
    world maps, never assume one.  The arm reaches about 0.8 m from the
    base: a target farther away in x/y needs ``navigate_to`` first.  Every
    base motion (``navigate_to``, ``move_base``, a VLA run) moves the arm and
    all three cameras with it, so re-localise before the next arm target.

    Gripper commands: ``robo.CLOSE`` (+1) closes and keeps squeezing,
    ``robo.OPEN`` (-1) opens, ``robo.HOLD`` (``"hold"``) servos the fingers to
    the width they had when the motion began.  ``"hold"`` is the default of
    every motion call and the carry-safe choice: a sustained +1 squeezes a
    small object out of the fingers, and carrying with -1 drops it.

    Perception is camera based: three 256x256 cameras ride on the robot,
    ``agentview`` (shoulder view: what is where), ``wrist`` (on the gripper:
    close-range geometry) and ``navview`` (forward-down floor view: where
    to drive), each with a per-pixel world map.  Object poses are never
    given.

    The frozen RLDX-1 VLA (``rldx_skill``, ``rldx_arm``) always receives the
    benchmark's own task instruction (``robo.task``); there is no prompt to
    write.  Success is the benchmark's own predicate (``state().success``,
    ``robo.done``); ``state().task_progress`` shows the values that predicate
    computes and ``success_criteria()`` its source.  Motion calls block until
    the primitive finishes and return a typed result; nothing is queued.

Constants:
    robo.OPEN = -1.0
    robo.CLOSE = 1.0
    robo.HOLD = 'hold'
    robo.MAX_MOVE_M = 0.3

Methods and attributes of `robo`:

robo.task -> str  (read-only attribute)
    The task instruction, e.g. ``"Open the left drawer."``.

    It is authoritative and the VLA acts on exactly this text.

robo.done -> bool  (read-only attribute)
    True once the benchmark's success predicate (``state().success``) has fired (sticky).

robo.state(step: int = -1) -> State
    Read the recorded state of the robot and of the episode.

    Args:
        step: recorded step to read; ``-1`` is the latest, ``0`` the
            initial scene.  A new step is recorded after every motion call.
    Returns:
        State with the gripper pose, finger positions, base pose, the
        success flag, ``task_progress`` and ``vla_desync``.

robo.image(camera: str = 'agentview', *, resolution: str = 'low', step: int = -1) -> Any
    Load a recorded camera image as a ``numpy`` uint8 array (256, 256, 3), RGB.

    Args:
        camera: ``"agentview"`` (shoulder camera: identify objects,
            fixtures and the layout; its pixels back-project directly),
            ``"wrist"`` (on the gripper: close-range geometry, best within
            20 cm of a target) or ``"navview"`` (forward-down view of the
            floor ahead of the base: where driving is possible).  All
            three move with the robot.
        resolution: there is one resolution, ``"low"`` (256x256).  Pixel
            coordinates are (row, col) with row 0 at the top and col 0 at
            the left; pass the same camera and step to
            ``back_project_batch``.
        step: recorded step, ``-1`` = latest.

robo.world_map(
    camera: str = 'agentview',
    *,
    resolution: str = 'low',
    step: int = -1,
) -> Any
    Load the per-pixel world coordinates of a recorded image.

    Returns a ``numpy`` float32 array (256, 256, 3): ``world_map[row, col]``
    is the [x, y, z] of the surface seen at that pixel (zeros where the
    camera saw no depth).  This is what ``back_project_batch`` and
    ``query_world_map`` read; load it to reason about many pixels at once
    (e.g. ``np.median(world_map[r0:r1, c0:c1, 2])`` is a surface height).
    Maps older than 25 recorded steps are deleted; a missing map raises
    ``ToolError``.

robo.show(camera: str = 'agentview') -> None
    Attach the current image of ``camera`` to the feedback of this turn.

    Images are not returned automatically; call ``show("agentview")``,
    ``show("wrist")`` or ``show("navview")`` when *you* (the author) need
    to look at the scene before writing the next turn.  Costs tokens:
    look only when a decision depends on it.

robo.artifact(name: str, *, step: int = -1) -> bytes
    Return the raw bytes of a recorded artifact (advanced).

    Names are ``agentview.png``, ``wrist.png``, ``navview.png``,
    ``navview_floor.png``, ``<camera>_world.npz`` (all three cameras),
    ``<camera>_depth.npz`` and ``<camera>_metadata.json`` (agentview and
    wrist only).

robo.depth(camera: str = 'agentview', *, step: int = -1) -> Any
    Load the metric depth of a recorded ``agentview`` or ``wrist`` image.

    Returns a ``numpy`` float32 array (256, 256) in metres.  ``navview``
    has no depth map (use its world map).

robo.floor_overlay(step: int = -1) -> Any
    Load the navview image with the floor painted green: uint8 (256, 256, 3), RGB.

    Floor = pixels whose world z lies between -0.2 and 0.12 m; green
    area ahead of the base is where it can drive.

robo.camera_meta(camera: str = 'agentview', *, step: int = -1) -> dict[str, Any]
    Return the camera calibration of a recorded step as a dict (``agentview`` / ``wrist``).

    Keys: ``intrinsic`` (3x3), ``extrinsic_cam2world`` (4x4), ``height``,
    ``width``, ``depth_near`` / ``depth_far``.  Both cameras move with the
    robot (the wrist camera with the gripper), so read the meta of the
    same step as the image.

robo.success_criteria() -> str
    Return the source code of this task's success predicate (``_check_success``).

    It names the objects and fixtures that matter and the thresholds
    (e.g. a drawer joint fraction of 0.95, a ``success_time`` counter)
    that ``state().task_progress`` reports live.  It contains no object
    coordinates.

robo.back_project_batch(
    pixels: Sequence[Sequence[int]],
    *,
    step: int = -1,
    camera: str = 'agentview',
    resolution: str = 'low',
) -> BackProjection
    Return the world [x, y, z] under image pixels of a recorded view.

    Args:
        pixels: 1 to 50 ``[row, col]`` pairs read off ``image(camera,
            step=step)`` (a single pair is accepted too).
        step: recorded step whose world map to read, ``-1`` = latest.
        camera: ``"agentview"``, ``"wrist"`` or ``"navview"``: the pixel
            coordinate space, so use the view the pixels came from.
        resolution: ``"low"`` (the only one recorded).
    Returns:
        BackProjection; ``median_xyz`` is the robust coordinate of what
        the pixels cover.  Every point is a visible SURFACE point (the
        top of an object, not its centre): sample 3-8 pixels firmly on
        the object's top surface and use the median; pixels on thin rims,
        edges, shadows or the gap to a counter hit the background.
    Raises:
        ToolError: no world map recorded for that step / camera.

robo.query_world_map(
    z_min: float = 0.85,
    z_max: float = 0.95,
    *,
    x_range: Sequence[float] | None = None,
    y_range: Sequence[float] | None = None,
    camera: str = 'agentview',
    resolution: str = 'low',
    min_cluster_size: int = 10,
) -> HeightQuery
    Find what lies inside a world-z band of the LATEST world map, as pixel clusters.

    Args:
        z_min / z_max: height band in metres.  0.85-0.95 finds objects on
            a counter; ``camera="navview"`` with 0.0-0.12 finds walkable
            floor.
        x_range / y_range: optional ``[min, max]`` world limits.
        camera: which camera's world map (``"agentview"``, ``"wrist"``,
            ``"navview"``).
        resolution: ``"low"`` (the only one recorded).
        min_cluster_size: smallest cluster reported (pixels).
    Returns:
        HeightQuery; each ``Cluster.center_xyz`` is a candidate location,
        ``sample_pixel`` a place to look in the image to identify it.
        Always reads the latest recorded step.

robo.rldx_skill(
    *,
    base_clip: float | None = None,
    max_chunks: int = 70,
    force_reset: bool = False,
    n_action_steps: int = 8,
    settle_patience: int = 999,
    settle_eps: float = 0.012,
) -> Skill
    Let the frozen RLDX-1 VLA drive the whole body (arm AND base) on the task instruction.

    The policy always receives the benchmark's own instruction
    (``robo.task``) and the three current camera images, executes
    ``n_action_steps`` env steps per chunk and stops when the task
    succeeds, when it settles or at ``max_chunks``.  It is the tool for
    grasps, re-grasps, pulling / pushing fixtures, pressing controls and
    other contact-rich motion, and it may reposition the base by itself.

    Args:
        base_clip: cap on the base velocity commands (None = full base
            motion; ``rldx_arm`` is the same skill with 0.1).
        max_chunks: action-chunk cap.  The evaluation protocol may pin it
            (and ``n_action_steps`` / ``settle_patience``) through the
            environment; ``Skill.max_chunks`` reports what applied.
        force_reset: start from a fresh frame history even after a
            previous VLA call (the runtime does this by itself whenever
            a non-VLA primitive ran in between: see
            ``state().vla_desync``).
        n_action_steps: env steps executed per predicted chunk.
        settle_patience: chunks without arm / gripper motion after which
            the run stops as ``settled``; leave the default (999 disables
            it: the VLA is expected to run until success or the cap).
        settle_eps: motion threshold of that test, metres.
    Returns:
        Skill; check ``status``, the grasp signals and ``robo.done``.
        ``cap`` means the VLA was still acting: call again (its frame
        history continues) rather than staging by hand.

robo.rldx_arm(
    *,
    base_clip: float | None = 0.1,
    max_chunks: int = 70,
    force_reset: bool = False,
    n_action_steps: int = 8,
    settle_patience: int = 999,
    settle_eps: float = 0.012,
) -> Skill
    Let the frozen RLDX-1 VLA drive the arm with the base clamped to small motions.

    Same skill and arguments as ``rldx_skill`` with ``base_clip=0.1``: the
    VLA can micro-align the gripper for a grasp or a contact but cannot
    drive away.  Use it once the base already stands at the right place;
    use ``rldx_skill`` when the VLA has to reposition the base as well.

robo.move_to(
    xyz: Sequence[float],
    gripper: float | str = 'hold',
    *,
    step_clip: float = 0.02,
    max_steps: int = 200,
    tol: float = 0.012,
) -> Move
    Servo the gripper to a world-frame position, holding its orientation.

    A closed-loop OSC servo: each env step moves the gripper at most
    ``step_clip`` towards the target until it is within ``tol`` or
    ``max_steps`` are spent.  The first arm servo after a base motion
    spends a few extra steps calibrating.

    Args:
        xyz: target gripper position [x, y, z] in metres.  One call may
            move at most 0.30 m (``ValueError`` beyond); split longer
            traversals into 2-3 waypoints at carrying height.
        gripper: ``"hold"`` (default: keep the finger width, carry-safe),
            +1 (close and keep squeezing) or -1 (open); held during the
            whole motion.
        step_clip: per-step travel cap (m): 0.02 in free space, 0.012 for
            a fine approach or a vertical retreat.
        max_steps: env step budget.
        tol: stop when within this distance (m).
    Returns:
        Move; check ``reached`` and compare ``final_eef_pos`` with the
        target: a stalled servo returns normally.

robo.move_delta(
    dxyz: Sequence[float],
    gripper: float | str = 'hold',
    *,
    step_clip: float = 0.02,
    max_steps: int = 80,
) -> Move
    Servo the gripper by a relative displacement from its current position.

    ``target = current eef_pos + dxyz`` (tolerance 0.012 m); for small
    adjustments such as a final approach or a lift.  ``|dxyz|`` may not
    exceed 0.30 m.  ``gripper`` as in ``move_to`` (default ``"hold"``).

robo.rotate_pitch(
    target_pitch: float = 0.6,
    *,
    gripper: float = 1.0,
    n: int = 12,
) -> Pitch
    Tilt the wrist by ``target_pitch`` radians (relative, about the controller's x axis).

    Pitches the gripper forward / back over ``n`` env steps while the
    position is held by the controller (it is not actively servoed:
    expect a few mm of drift).  Use it before threading the gripper into
    an opening whose front face points along world +/-y.  ``gripper`` is
    a numeric command held during the rotation (+1 keep closed, -1 open;
    ``"hold"`` is not accepted here).  Read ``state().eef_tilt``
    afterwards; the primitive reports no angle.

robo.set_gripper(gripper: float = 1.0, steps: int = 10) -> Gripper
    Hold the gripper pose and drive the fingers with ``gripper`` for ``steps`` env steps.

    ``set_gripper(+1, steps=10)`` closes (or firms a grip before a
    carry), ``set_gripper(-1)`` opens.  Numeric only: ``"hold"`` is not a
    command here.  Verify a grasp afterwards from
    ``Gripper.gripper_opening`` (clearly above 0 = something between the
    fingers) and the wrist image.

robo.release(steps: int = 10) -> Gripper
    Open the gripper in place for ``steps`` env steps (``set_gripper(-1, steps)``).

    Release only when the object is supported at its destination, then
    retreat upwards and check that it stayed put.  ``terminated``
    reports whether letting go satisfied the task.

robo.scripted_grasp(
    xyz: Sequence[float],
    *,
    approach_z: float = 0.1,
    grasp_z_offset: float = 0.0,
    step_clip: float = 0.02,
) -> Grasp
    Coarse scripted grasp: open, hover ``approach_z`` above ``xyz``, descend, close, lift.

    A fallback for simple, well-localised objects; prefer ``rldx_arm``
    for anything hard.  ``grasp_z_offset`` shifts the closing height
    (negative = below the sampled surface point); ``step_clip`` caps the
    descent speed.  The hover point must be within 0.30 m of the gripper.
    Returns Grasp; ``ok`` False names the ``stage`` that stalled.

robo.navigate_to(
    xy: Sequence[float],
    *,
    tol: float = 0.2,
    max_steps: int = 300,
    gripper: float | str = 'hold',
) -> Navigation
    Drive the base towards a world (x, y) and stop facing it.

    The base first turns to face the target, then drives forward at full
    speed (about 2.3 mm per env step, so 300 steps cover roughly 0.7 m)
    with closed-loop steering until it is within ``tol``; the first call
    spends a few steps calibrating the heading.  There is no path
    planning: choose a target with a clear line ahead (``floor_overlay``,
    ``query_world_map`` on the navview) and expect ``stuck`` when a
    fixture is in the way.  The arm is held in place and its world
    position changes with the base.

    Args:
        xy: world target [x, y] in metres (a third value is ignored).
        tol: stopping distance (m): the standoff you want in front of a
            fixture plus the object's half depth (``tol=0.6`` stops about
            0.6 m in front of the target).
        max_steps: env step budget; increase for a long drive or call
            again.
        gripper: ``"hold"`` (default, carry-safe), +1 or -1 while driving.
    Returns:
        Navigation; check ``reached`` / ``stuck`` and re-localise
        everything afterwards.

robo.move_base(
    forward: float = 0.0,
    lateral: float = 0.0,
    turn: float = 0.0,
    *,
    steps: int = 10,
    gripper: float | str = 'hold',
) -> BaseMove
    Drive the base with raw velocities in its own frame for ``steps`` env steps.

    ``forward`` (+ = ahead), ``lateral`` (+ = strafe right) and ``turn``
    (+ = rotate counter-clockwise) are clipped to [-1, 1].  For fine
    adjustments near a fixture (a short nudge, a small turn): keep
    ``forward <= 0.4``, ``turn <= 0.3`` and ``steps <= 20`` per call,
    then re-observe.  ``gripper`` as in ``navigate_to``.
    Returns BaseMove with the measured world displacement of the base.

Result types (frozen dataclasses; access fields as attributes):

class State(
    step: int,
    task: str,
    eef_pos: tuple[float, float, float],
    eef_quat: tuple[float, float, float, float],
    gripper_qpos: tuple[float, float],
    gripper_opening: float,
    base_pos: tuple[float, float, float],
    base_quat: tuple[float, float, float, float],
    success: bool,
    task_progress: dict[str, Any],
    vla_desync: bool,
    terminated: bool,
    truncated: bool,
)
    Snapshot of the robot and of the episode at one recorded step.

    step: index of the recorded step (0 = initial scene).  A new step is
        recorded after every motion call.
    task: the task instruction given by the benchmark (authoritative; the VLA
        acts on exactly this text).
    eef_pos: gripper (end-effector) position [x, y, z] in the world frame,
        metres.  This is what ``move_to`` servos and what ``back_project_batch``
        coordinates are directly comparable to.
    eef_quat: gripper orientation as a unit quaternion [x, y, z, w].
    gripper_qpos: the two finger joint positions (q0 >= 0, q1 <= 0).
    gripper_opening: finger separation proxy |q0| + |q1|: about 0.04 at the
        start (fingers half open), about 0.08 fully open, about 0.0 closed on
        nothing; a value that stays in between after closing means the
        fingers stopped on something.
    base_pos: mobile-base position [x, y, z] in the world frame, metres.
    base_quat: base orientation [x, y, z, w]; ``base_yaw`` is its heading.
    success: the benchmark's own success predicate (``_check_success``) has
        fired.  This is the only definition of success; it equals
        ``terminated`` and ``robo.done``.
    task_progress: the live values the success predicate computes (its
        counters, flags and sub-predicates, e.g. a joint fraction, a
        ``success_time`` counter, ``is_open`` flags).  The keys are
        task-specific: read them at step 0, then compare before and after an
        action to see whether the action moved the criterion.  May be empty.
    vla_desync: a non-VLA primitive ran since the last VLA call, so the next
        ``rldx_skill`` / ``rldx_arm`` starts from a fresh frame history.
    terminated: same as ``success``.
    truncated: RPent never reports a step budget for this robot (always
        False); the host's wall-clock and decision budgets end an episode.
    .base_yaw -> float: Heading of the base about world z in radians (0 = facing +x, pi/2 = facing +y).
    .eef_approach -> tuple[float, float, float]: Unit vector the fingers point along, world frame ((0, 0, -1) = straight down).
    .eef_tilt -> float: Angle between the approach axis and straight down, radians (0 = down, pi/2 = level).

class Move(
    target_xyz: tuple[float, float, float],
    final_eef_pos: tuple[float, float, float],
    final_dist_m: float,
    reached: bool,
    steps_used: int,
    max_steps: int,
    gripper_q0: float,
    terminated: bool,
)
    Outcome of ``move_to`` / ``move_delta`` (a closed-loop OSC servo).

    target_xyz: the commanded world target.
    final_eef_pos: where the gripper actually stopped (metres).
    final_dist_m: distance between the two.
    reached: ``final_dist_m < tol``.  False means the servo used its whole
        step budget without converging (out of reach, contact, joint limit):
        re-plan (navigate closer, change the approach) instead of repeating it.
    steps_used / max_steps: env steps consumed / allowed.
    gripper_q0: first finger joint after the move (RPent's gripper width
        proxy: about 0.02 half open, about 0.0 closed on nothing).
    terminated: the benchmark's success predicate after the move
        (same as ``robo.done``).

class Pitch(
    target_pitch: float,
    final_eef_pos: tuple[float, float, float],
    steps_used: int,
    terminated: bool,
)
    Outcome of ``rotate_pitch``.

    No measured angle is reported by the primitive: read
    ``state().eef_tilt`` / ``state().eef_approach`` afterwards.

class Gripper(
    gripper: float,
    steps: int,
    gripper_qpos: tuple[float, float],
    gripper_opening: float,
    terminated: bool,
)
    Outcome of ``set_gripper`` / ``release``.

    gripper: the command that was held (+1 close, -1 open); steps: for how
        many env steps.  ``gripper_qpos`` / ``gripper_opening`` are the finger
        positions afterwards (a position, not proof of a hold: verify a
        grasp from the wrist image and from the object moving with the
        gripper).

class Grasp(
    ok: bool,
    stage: str | None,
    final_eef_pos: tuple[float, float, float],
    final_dist_m: float,
    gripper_qpos: tuple[float, float],
    gripper_opening: float,
    terminated: bool,
)
    Outcome of ``scripted_grasp`` (open, hover, descend, close, lift).

    ok: every stage servoed to its target.  ``stage`` names the stage that
        stalled (``approach`` | ``descent`` | ``lift``) or is None when ok;
        ``final_dist_m`` is that stage's remaining distance (0 when ok).
    final_eef_pos: where the gripper ended (after the lift when ok).
    gripper_opening: finger separation afterwards; a value clearly above 0
        after the close means the fingers stopped on something.

class Navigation(
    target_xy: tuple[float, float],
    reached: bool,
    stuck: bool,
    final_dist_m: float,
    moved_m: float,
    steps_used: int,
    max_steps: int,
    start_xy: tuple[float, float],
    base_pos: tuple[float, float, float],
    terminated: bool,
)
    Outcome of ``navigate_to`` (drive the base towards a world x/y).

    reached: the base came within ``tol`` of the target and faces it.
    stuck: it ran out of steps having moved less than 0.12 m (it rammed a
        fixture: there is no path planning).  Back off with ``move_base``
        and approach from another side.
    final_dist_m: remaining distance to the target; moved_m: distance
        travelled from ``start_xy``; base_pos: the base afterwards.
    steps_used / max_steps: env steps consumed / allowed.

class BaseMove(
    base_moved: tuple[float, float, float],
    base_pos: tuple[float, float, float],
    steps: int,
    terminated: bool,
)
    Outcome of ``move_base`` (raw base velocities in the robot's local frame).

    base_moved: world displacement [dx, dy, dz] of the base over the call;
    base_pos: the base afterwards; steps: env steps executed.

class Skill(
    status: str,
    chunks_used: int,
    steps_applied: int,
    max_chunks: int,
    grasped: bool,
    grasp_detected: bool,
    grasp_contact: bool,
    held_apart: bool,
    grasp_obj: str | None,
    gripper_q0: float,
    peak_lift_m: float,
    base_drift_m: float,
    base_clip: float | None,
    instruction: str,
    terminated: bool,
)
    Outcome of ``rldx_skill`` / ``rldx_arm`` (a closed-loop RLDX-1 run).

    status: why the run stopped: ``success`` (the benchmark predicate fired),
        ``settled`` (arm and gripper stopped changing for ``settle_patience``
        chunks) or ``cap`` (``max_chunks`` reached; the VLA was still acting:
        call again to continue with its frame history intact).
    chunks_used / steps_applied: action chunks and env steps executed.
    max_chunks: the chunk cap that applied (the evaluation protocol may pin
        it through the environment, overriding the argument).
    grasped: holding something at the end of the call (carry-ready).
    grasp_detected: held something at some chunk (it may have been placed
        or dropped since).
    grasp_contact: both fingerpads touch the same task object right now (the
        benchmark's own grasp check; works for any grasp direction).
    held_apart: the VLA commanded a close and the fingers stopped apart, i.e.
        something is between them (also a fixture handle, which
        ``grasp_contact`` cannot see).  grasp_obj: the contacted object's
        name, or None.
    gripper_q0: first finger joint at the end; peak_lift_m: highest rise of
        the gripper above its lowest point during the run; base_drift_m: how
        far the base moved; base_clip: the base-motion cap that applied.
    instruction: the task text the policy received (always ``robo.task``).
    terminated: the benchmark's success predicate after the run.

class BackProjection(
    points: tuple[Point, ...],
    median_xyz: tuple[float, float, float] | None,
    valid_count: int,
    total_count: int,
    camera: str,
    resolution: str,
    step: int,
)
    Outcome of ``back_project_batch``: one ``Point`` per requested pixel, in order.

    median_xyz: per-axis median of the valid points (None when none are
        valid): the robust object coordinate when the pixels lie on one
        object's top surface.  valid_count / total_count: how many pixels
        had a world coordinate.  step / camera: the world map that was read.
    .xyz -> tuple[tuple[float, float, float], ...]: The valid world coordinates only, in request order.

class Point(
    pixel: tuple[int, int],
    world_xyz: tuple[float, float, float] | None,
    error: str | None,
)
    One pixel of ``back_project_batch``: its world coordinate or why it has none.

    world_xyz: [x, y, z] of the surface seen at ``pixel`` (metres), None when
        the pixel is out of bounds or has no valid depth (``error`` says
        which).
    .valid -> bool: True when ``world_xyz`` is available.

class HeightQuery(
    clusters: tuple[Cluster, ...],
    total_pixels: int,
    camera: str,
    resolution: str,
)
    Outcome of ``query_world_map``: clusters of pixels inside a world-z band.

    clusters: largest first (at most 20); total_pixels: pixels that matched
        the band and the optional x/y ranges before clustering.  Clustering
        is a coarse image grid, not connected components: one object can
        span two clusters and two touching objects can share one.

class Cluster(
    center_xyz: tuple[float, float, float],
    bbox_min: tuple[float, float, float],
    bbox_max: tuple[float, float, float],
    pixel_count: int,
    sample_pixel: tuple[int, int],
)
    One cluster of ``query_world_map``: pixels of one grid cell inside the height band.

    center_xyz: median world point of the cluster; bbox_min / bbox_max: its
        world extent; pixel_count: matched pixels; sample_pixel: one
        ``(row, col)`` inside it (look there in the image to see what it is).

Exceptions:
    ToolError(RuntimeError): a call was rejected or a service failed; catch it if you have a fallback.
    ValueError: an argument was invalid (e.g. a planar move over 0.30 m) - nothing was executed.
    EpisodeFinished(BaseException): the episode is over (reason: terminated | truncated | timeout | cancelled); return.