RoboColiseum

RoboColiseum is an embodied-AI benchmark jointly initiated by nearly 20 universities, companies and research institutes worldwide, including Hugging Face, NVIDIA and Tsinghua University. It runs on GenieSim: the G2 robot in Isaac Sim, on four boards (instruction following, spatial reasoning, general manipulation and robustness). GenieSim's own benchmark runs the simulator and judges every episode, and asks a policy for actions over its inference protocol; PyRUA-Lean answers it as that policy. A language model writes Python against robo, one cell at a time, while the episode runs. No VLA policy, no privileged information.

Overview

Robot	AgiBot G2 (G2_omnipicker): two 7-joint arms with one gripper each, a 5-joint waist and a wheeled base
Agent runtime	openai: any OpenAI-compatible chat endpoint (GPT-6.1 Sol, reasoning effort high, in our runs)
Models on the policy side	SAM3 (segmentation), as one HTTP service for every policy process; no VLA
Boards	instruction, spatial, manip, robust
Hardware	Linux x86_64, Python 3.11 or 3.12 (the arm IK's wheels); an NVIDIA GPU for SAM3 (about 10 GB). The simulator runs on the platform, or wherever you run GenieSim

Observation and Action

Item	Description
Observation	GenieSim's inference request: three colour images (head, hand_left, hand_right), each camera's depth image and calibration (intrinsics and pose; the benchmark withholds the calibration while it degrades the images, as the robustness board's camera-quality cases do), the joints of both arms, the grippers and the waist, both end-effector poses, and the instruction. No object poses, and the task's progress is not read.
Action	One action per request: arm motions are solved in the library on the benchmark's own IK (ik_solver) and sent as joint targets (JOINT_ABS), with the gripper commands; the waist as joint targets, the base as a relative pose.
Success	The benchmark's own checks of the scene, stage by stage, every 30 actions. The agent never sees them; when a program finishes, the robot holds still until the benchmark ends the episode.
Task prompt	The instruction, and one knowledge text for every task: how the tools behave and how the benchmark judges a stage (which state counts, when it is checked). No task recipes.

Installation and Resources

1. The policy environment, from a PyRUA-Lean checkout:

   pip install -e ".[robocoliseum]"
   

   The extra installs the bridge's and perception's packages and the benchmark's arm IK, ik_solver, which is not on PyPI: its wheel comes from GenieSim's 3rdparty/ at a fixed commit. The robot's URDF and the solver's configuration are GenieSim's own, shipped in pyrualean/hosts/robocoliseum_ik/; GENIESIM_IK_SDK names another directory holding them (G2_NO_GRIPPER.urdf, g2_solver.yaml) when GenieSim's robot description changes.

2. SAM3, as one service for every policy process on the machine. Its environment needs a CUDA build of torch, the sam3 package (it pins numpy<2, so a separate environment is simplest) and the checkpoint:

   pip install "sam3 @ git+https://github.com/RLinf/sam3.git@c1f6b1e83967c912f675d8975d437ccdc41e17d8" \
       einops pycocotools psutil
   bash integrations/xpolicylab/PyRUA_Lean/download_assets.sh   # sam3.pt (3.4 GB), into that directory's checkpoints/
   python src/pyrualean/hosts/sam3_http.py --port 8790 \
       --checkpoint integrations/xpolicylab/PyRUA_Lean/checkpoints/sam3.pt
   

   It prints SAM3 ready on http://127.0.0.1:8790 once loaded. PYRUA_SAM3_ENDPOINT points the policy processes at another address. Without the service, finding an object by name reports that SAM3 is not running; measuring the object at a pixel still works.

Model Configuration

The cells are written by a model behind an OpenAI-compatible chat endpoint, with play's options:

Option	Our runs	Meaning
--runtime openai		the in-process agent loop
--base-url, --api-key-env, --model	the endpoint, the variable holding its key, gpt-6.1-sol
--openai-extra-body	'{"reasoning_effort": "high"}'	JSON merged into every request
--openai-max-tokens	32000	the longest reply
--images	on-demand	the model sees a camera only when its code asks for it (robo.show)
--budget-s	1800	wall-clock limit of an episode, in seconds

Run a Task

Online (RoboColiseum)Your own GenieSim run

Submit a job for one board on the platform first; it gives the job's uuid, its tunnel endpoint, a job token and its parallelism. Each agent process dials the platform's gateway and serves one tunnel: start up to parallelism of them, and they play the cases the platform sends until it drains the job.

export JOB_UUID=... TUNNEL_ENDPOINT=ws://.../api/challenge/tunnel JOB_TOKEN=...
export OPENAI_API_KEY=...
for i in $(seq 1 "$PARALLELISM"); do
  python -m pyrualean.hosts.geniesim_robocoliseum agent --arm cells --out "runs/$JOB_UUID/agent_$i" \
      --runtime openai --base-url https://YOUR_API_HOST/v1 --api-key-env OPENAI_API_KEY \
      --model gpt-6.1-sol --openai-extra-body '{"reasoning_effort": "high"}' \
      --openai-max-tokens 32000 --images on-demand --budget-s 1800 &
done
wait

An agent exits 0 once the job is drained, 1 when its tunnel is refused or stays lost past the gateway's 30-second reconnect window, 130 on Ctrl-C. The platform keeps the scores.

Start the policy first, then the benchmark with its inference host pointing at it:

python -m pyrualean.play --robot robocoliseum --arm cells --out runs/pick_block_color \
    --task g2op_if_pick_block_color --port 8999 --output-root <genie_sim>/output \
    --runtime openai --base-url https://YOUR_API_HOST/v1 --api-key-env OPENAI_API_KEY \
    --model gpt-6.1-sol --openai-extra-body '{"reasoning_effort": "high"}' \
    --openai-max-tokens 32000 --images on-demand

# in GenieSim's simulator environment, from its checkout
python source/geniesim_benchmark/src/geniesim_benchmark/app/app.py \
    --config source/geniesim_benchmark/src/geniesim_benchmark/config/g2op_if_pick_block_color.yaml \
    --benchmark.infer_host 127.0.0.1:8999 --benchmark.model_arc corobot

play plays the run's first episode and ends the rest. --output-root, the benchmark's output directory, lets it read the episode's scores.

View Results

• Each episode's card, prompt, programs, tool calls, transcript and every recorded step (the three images and the state) are kept under --out: episode_NNN/ of each session for an agent, the directory itself for play.
• An agent appends one line per episode to online.jsonl (session, task, instruction, status, actions).
• With --output-root, benchmark_verdict.json holds the benchmark's scores of the episode: one per stage (STEPS) and E2E.

What Runs Where

• Benchmark (GenieSim, in Isaac Sim: on the platform online, or in your simulator environment) owns the simulator: it builds every request, counts each stage's action budget and judges the episode.
• Policy process (this library) answers the benchmark's requests: as a WebSocket server the benchmark connects to, or online through the platform's tunnel, which it dials. It runs the agent loop, the sandboxed cells and the primitives (inverse kinematics, gripper, perception).
• SAM3 service segments for every policy process on the machine.
• Model endpoint writes the cells.

One request is answered with one action, so a primitive sees the result of every native action. The benchmark does not step the simulator while the model thinks.

Primitives the Agent Can Call

Motion:

• move_to(xyz, arm, gripper, pitch, yaw, roll, path): move one arm's grasp point to a position, with an orientation. A pose out of reach moves nothing, and the result names reachable poses nearby.
• move_both(left, right): both arms' grasp points at once, each to a position.
• set_gripper(arm, command): close or open one gripper until the fingers stop; the result says whether something is held.
• move_waist(forward, up, turn_deg, lean_deg), move_base(forward, left, turn_deg): the waist and the wheeled base.
• home(arm="both"): return an arm to the joint angles it had when the episode started.
• wait_for_check(checks=1): hold still until the benchmark's next stage check has seen the scene.

Perception and state:

• locate(target, camera): find an object by words (SAM3) or at a pixel and measure it in 3D from the depth image: centre, top, the surface it stands on, extent, tilt, and the yaw of a top grasp across it.
• segment(camera, text=..., row=..., col=...): the object's region in the current image.
• back_project(row, col, camera): the point seen at one pixel.
• state(), image(camera), show(camera), artifact(name): the recorded state of both arms, the images, and what the model looks at.

API reference: what the agent reads, generated from pyrualean.robocoliseum when these docs are built

class RobocoliseumRobot
    The G2 dual-arm humanoid with two-finger grippers, in the GenieSim benchmark.

    Frame: ``robo.FRAME`` is ``base_link``, fixed to the robot's base: +x forward, +y the
    robot's left, +z up; metres and degrees.  The upper body may start turned on its waist:
    ``state().torso_heading_deg`` says where the chest (and the work surface) faces.  The
    arms are ``"left"`` and ``"right"``; a motion names one arm and the other holds.

    Orientation is three angles (``state().right.pitch``, ``.yaw``, ``.roll``): ``pitch`` 0
    points level, 90 straight down; ``yaw`` is the heading of the pointing direction, 0 along
    +x, +90 towards +y; ``roll`` turns the fingers about the pointing direction (0: they close
    horizontally).  Pointing straight down, ``yaw = locate(...).grasp_yaw`` closes the fingers
    across an object's narrow side.

    ``move_to`` drives the GRASP POINT (the gripper's centre, 0.143 m out from the wrist
    flange): it solves the arm's inverse kinematics for the whole pose, refuses one out of
    reach without moving (and names reachable ones), and follows a straight line
    (``path="line"``) or a joint swing.

    Gripper: COMMANDS are ``robo.CLOSE``, ``robo.OPEN`` and ``robo.KEEP`` (None);
    MEASUREMENTS (``state().right.opening``, ``Gripper.opening``) are openings, 1.0 open and
    0.0 closed on nothing.

    Perception: ``locate("words")`` finds an object by description and measures it in 3D from
    the depth image; ``locate(row, col)`` measures the object at a pixel.  Object poses are
    never given.  Every native action counts against the stage's budget (on the card when
    known).

Constants:
    robo.ARMS = ('left', 'right')
    robo.LEFT = 'left'
    robo.RIGHT = 'right'
    robo.CLOSE = 0.0
    robo.OPEN = -0.785
    robo.KEEP = None
    robo.FRAME = 'base_link'

Methods and attributes of `robo`:

robo.task -> str  (read-only attribute)
    The task instruction given by the benchmark (authoritative).

robo.state(step: int = -1) -> State
    Read the recorded state of both arms.

    Args:
        step: recorded step to read; ``-1`` is the latest, ``0`` the initial scene.
    Returns:
        State with both arms' tool centres, pointing directions and gripper openings.

robo.image(camera: str = 'head', *, resolution: str = 'high', step: int = -1) -> Any
    Load a camera image as a ``numpy`` uint8 array (H, W, 3), RGB.

    Args:
        camera: ``"head"`` (the whole scene), ``"hand_left"`` or ``"hand_right"`` (on
            the gripper). Pixel coordinates are (row, col), row 0 at the top.
        resolution: there is one resolution per camera, ``"high"``.
        step: recorded step, ``-1`` = latest.

robo.show(camera: str) -> None
    Attach the current image of ``camera`` to the feedback of this turn.

    Images are not returned automatically: call ``show(<camera>)`` when
    *you* (the author) need to look at the scene before writing the next
    turn.  Costs tokens: look only when a decision depends on it.

robo.artifact(name: str, *, step: int = -1) -> bytes
    Return the raw bytes of a recorded artifact (advanced).

    Names are ``head.png``, ``hand_left.png`` and ``hand_right.png``.

robo.move_to(
    xyz: Sequence[float] | None = None,
    arm: str = 'right',
    gripper: float | None = None,
    pitch: float | None = None,
    yaw: float | None = None,
    roll: float | None = None,
    path: str = 'auto',
    max_actions: int = 60,
) -> Move
    Move one arm's grasp point to a position in ``robo.FRAME``, with an orientation.

    The arm's inverse kinematics is solved for the whole pose first (the same solver
    the benchmark uses), and the arm follows a planned path commanded as joint angles.
    Nothing moves when the pose is out of reach: ``stop_reason == "out_of_reach"``,
    ``steps_used == 0``, and ``alternatives`` names nearby poses that ARE reachable.
    A pose missed by at most 2 cm / 8 degrees is taken anyway: the arm goes to the
    closest one and ``note`` says how far off it is.

    Args:
        xyz: target grasp point [x, y, z], metres; None keeps the position.
        arm: ``"left"`` or ``"right"``.  The other arm holds its joints.
        gripper: ``robo.CLOSE`` / ``robo.OPEN`` to command the moving arm's gripper on
            every action, ``robo.KEEP`` (None) to keep commanding what it was last told
            (a held object stays squeezed).
        pitch / yaw / roll: the orientation to arrive with, degrees (the class
            docstring defines them).  An angle left None keeps its current value, so
            a plain position move does not turn the gripper.
        path: ``"auto"`` (a straight line of the grasp point where it can be followed,
            else a joint swing), ``"line"`` (the line or nothing: every approach,
            descent and lift near objects) or ``"joint"`` (swing the joints: fewer
            actions, the grasp point does not follow a line — away from objects only).
        max_actions: native actions this call may spend.
    Returns:
        Move; test ``reached`` (within 0.01 m and 5 degrees).
    Raises:
        ValueError: a malformed argument (nothing is sent).

robo.set_gripper(
    arm: str = 'right',
    command: float | None = None,
    wait_for_check: bool = True,
) -> Gripper
    Close or open one gripper, without moving the arms.

    The command is repeated for a few actions until the fingers stop.  After a close,
    ``holding`` tells whether something stopped them apart.  A close that holds something
    then keeps the robot still for up to 30 more actions, until the benchmark's progress
    check has seen the grasp: a lift only counts when it starts after that check.  Later
    motions with ``robo.KEEP`` keep commanding what was commanded last, so a held object stays
    squeezed.

    Args:
        arm: ``"left"`` or ``"right"``.
        command: ``robo.CLOSE`` or ``robo.OPEN``.  ``robo.KEEP`` is refused: this
            call exists to change the gripper.
        wait_for_check: after a close that holds something, keep still until the
            progress check has seen it (up to 30 actions).  Keep True when the next
            stage is graded on LIFTING the object (a pick-up).  False skips the wait:
            for a re-grasp, or a grasp whose next stage is not judged on the lift.
    Returns:
        Gripper.  ``holding`` is not proof that the INTENDED object is held: lift,
        then look at the wrist camera or ``locate`` it again.
    Raises:
        ValueError: a malformed argument (nothing is sent).

robo.move_both(
    left: Sequence[float] | None = None,
    right: Sequence[float] | None = None,
    max_actions: int = 60,
) -> BothMove
    Move both grasp points at the same time — to carry one object with two hands.

    Each tool keeps its orientation and each gripper its last command.  The two hands
    cover the same fraction of their remaining way on every action, so they start and
    arrive together and keep their offset in between (move them by the same vector to
    carry something rigid).  None holds that arm.

    Args:
        left / right: target grasp point [x, y, z] in ``robo.FRAME``, or None.
        max_actions: native actions this call may spend.
    Returns:
        BothMove; ``reached`` when every moved arm arrived (1.5 cm).
    Raises:
        ValueError: a malformed argument (nothing is sent).

robo.move_base(
    forward: float = 0.0,
    left: float = 0.0,
    turn_deg: float = 0.0,
) -> BaseMove
    Drive the wheeled base — only in tasks where the base is free (the ``agent_*``
    scenes); elsewhere it is fixed to the floor and nothing moves.

    ``forward`` / ``left`` are metres along the base's own axes (``robo.FRAME`` x / y),
    ``turn_deg`` counter-clockwise.  At most 2 m and 180 degrees per call.  The arms keep
    their joints.  ``robo.FRAME`` travels with the base, so arm poses stay valid, but
    every object position measured before the move is stale: look again.

    Returns:
        BaseMove.  Open loop — the benchmark reports no base pose.

robo.move_waist(
    forward: float = 0.0,
    up: float = 0.0,
    turn_deg: float = 0.0,
    lean_deg: float = 0.0,
    reset: bool = False,
    max_actions: int = 60,
) -> Waist
    Move the upper body: the chest forward/back and up/down, or turn it.

    The G2 stands on a knee-hip-waist chain.  This moves the chest by ``forward`` and
    ``up`` metres in ``robo.FRAME`` while keeping the torso's lean (``lean_deg`` adds to
    it), and turns the upper body by ``turn_deg`` (+ to the robot's left).  The arms keep
    their joint angles, so BOTH hands — and anything held — move along with the chest.
    Keep the waist still by default and reach with the arms: move it only when an arm
    cannot reach (``move_to`` returned ``reach_hint``), because every hand position and
    every earlier measurement relative to the hands changes, and a waist move costs 10-30
    actions.  ``forward`` and ``up`` move the chest along robo.FRAME +x and +z — the plane
    the legs bend in — whichever way the torso faces; when the torso is turned (see
    ``state().torso_forward``), ``forward`` moves the chest sideways relative to it.
    ``reset=True`` returns to the starting waist.

    Args:
        forward / up: chest displacement, metres (the chest can come forward about
            0.3 m; going straight down is limited, forward-and-down goes further).
        turn_deg: upper-body turn, degrees.
        lean_deg: extra forward lean of the torso, degrees (tilts both arms with it).
        reset: back to the waist angles the episode started with.
        max_actions: native actions this call may spend (a move takes 10-30).
    Returns:
        Waist; when ``moved`` is False the pose is out of range and nothing moved.
        Re-read ``state()`` afterwards: every hand position has changed.

robo.wait_for_check(checks: int = 1) -> int
    Hold both arms and grippers still until the benchmark's stage check has run.

    The benchmark looks at the scene only once every 30 native actions, and a stage
    passes only on what it sees at such a check (some stages need two checks in a row).
    A state that is reached and then left between two checks — an object lifted and put
    down again — is never seen.  Call this right after reaching what a stage asks, before
    the next motion changes it: it spends at most 30 actions per check.

    Args:
        checks: how many checks to wait for (1-5); 2 for a stage that needs the state
            on two checks in a row (putting inside, stacking, beside).
    Returns:
        the native actions spent.

robo.home(arm: str = 'both', max_actions: int = 25) -> Home
    Return one arm or both to the joint angles the episode started with.

    Args:
        arm: ``"left"``, ``"right"`` or ``"both"``.
        max_actions: native actions this call may spend.
    Returns:
        Home; ``reached`` says whether every joint came within 0.02 rad of its start
        angle, ``max_joint_error_rad`` what remains.
    Raises:
        ValueError: a malformed argument (nothing is sent).

robo.segment(
    camera: str = 'head',
    *,
    row: int | None = None,
    col: int | None = None,
    text: str | None = None,
    min_score: float = 0.2,
) -> Segment
    Find one object's region in a camera image: by name, or by a pixel inside it.

    Args:
        camera: ``"head"``, ``"hand_left"`` or ``"hand_right"``.
        text: what to find, in a few words ("the black pen", "orange juice bottle"):
            SAM3 segments it, and its best mask scoring at least ``min_score`` is
            returned.  Name one object; a colour or position in the words helps.
        row / col: a pixel INSIDE the region to find; both None (and no text) takes
            the largest colour region in the image.
    Returns:
        Segment; when ``found``, pass ``centroid`` to ``back_project`` with the same
        camera.
    Raises:
        ValueError: an unknown camera, or only one of row / col.

robo.back_project(row: int, col: int, camera: str = 'head') -> WorldPoint
    Return the point seen at one pixel of the current image of ``camera``.

    Args:
        row / col: pixel, row 0 at the top, col 0 at the left.
        camera: the camera the pixel was read from (default ``"head"``).
    Returns:
        WorldPoint; ``xyz`` is where the camera ray meets the first surface.  A pixel
        on an edge or a thin part lands behind the object: prefer a region's centroid.
    Raises:
        ToolError: pixel out of range, no valid depth there, or no calibration.

robo.locate(
    target: str | int | None = None,
    col: int | None = None,
    camera: str = 'head',
    *,
    row: int | None = None,
    min_score: float = 0.2,
    near: tuple[int, int] | None = None,
) -> ObjectPose
    Measure one object in 3D: name it, or point at a pixel inside it.

    ``robo.locate("the black pen")`` — SAM3 finds what the words describe in the image
    and the depth image measures it.  ``robo.locate(row, col)`` — the object at that
    pixel, cut out by its shape above the surface it stands on (then SAM3's mask at the
    pixel, then the pixel's colour region).  Naming is the way for thin, small, or
    same-coloured objects and for objects inside containers; check ``method`` and
    ``score`` and the returned ``pixel_uv`` against the image.

    Args:
        target: what to find, in a few words (a string), or the pixel's row (an int,
            with ``col``).  None with no pixel takes the largest colour region.
        col: the pixel's column, with an int ``target`` (or ``row=``).
        camera: ``"head"`` (default) or a wrist camera; the head camera sees the
            whole scene, a wrist camera is close up.
        min_score: the lowest SAM3 confidence accepted for a named object.
        near: (row, col) — with a name, measure the instance of it at (or nearest)
            that pixel rather than the best-scoring one.  SAM3 ranks by what a thing
            is, not where: "the juice in the basket" can score a bottle on the shelf
            highest.  Read ``candidates``, or the image, and pass the one you mean.
    Returns:
        ObjectPose.  To grasp: hop to ``[x, y, top_z + 0.06]`` with
        ``gripper=robo.OPEN`` (keeping the orientation), descend to
        ``top_z - 0.02``, then ``set_gripper(arm, robo.CLOSE)``.
    Raises:
        ValueError: an unknown camera, or only one of row / col.

Result types (frozen dataclasses; access fields as attributes):

class State(
    step: int,
    task: str,
    left: ArmState,
    right: ArmState,
    torso_heading_deg: float,
    torso_forward: tuple[float, float, float],
)
    Snapshot of both arms at one recorded step.

    step: index of the recorded step (0 = the scene the episode started in).  A new step
        is recorded after every tool call that acted.
    task: the task instruction the benchmark generated (authoritative).
    left / right: the two arms.
    torso_heading_deg / torso_forward: where the upper body faces.  ``robo.FRAME`` is fixed
        to the robot's base and some tasks start with the torso TURNED (the warehouse ones
        by 90 degrees): the table is then along ``torso_forward``, not along +x.
        ``point="forward"`` / ``"left"`` / ``"right"`` follow the torso.

class ArmState(
    tool_centre: tuple[float, float, float],
    pointing: tuple[float, float, float],
    finger_axis: tuple[float, float, float],
    flange: tuple[float, float, float],
    opening: float,
    pitch: float | None,
    yaw: float | None,
    roll: float | None,
)
    One arm at a recorded step.

    tool_centre: [x, y, z] of the middle of the jaw plane, metres, in ``robo.FRAME``
        (``base_link``): this is what ``move_to`` drives.
    pointing: the unit direction the tool points, in ``robo.FRAME``.
    finger_axis: the unit line the fingers close along.
    pitch / yaw / roll: the tool's orientation in degrees, as ``move_to`` takes it.
    flange: position of the end-effector link (the wrist flange), metres.
    opening: measured gripper opening, NORMALISED here: 1.0 = open, 0.0 = closed.
        (``set_gripper``'s COMMANDS use the library's convention, see the class
        docstring.)

class Move(
    arm: str,
    target_tool_centre: tuple[float, float, float],
    final_tool_centre: tuple[float, float, float],
    dist_to_target_m: float,
    pointing_error_deg: float,
    reached: bool,
    steps_used: int,
    steps_planned: int,
    truncated: bool,
    reach_hint: str | None,
    rotation_error_deg: float,
    stop_reason: str | None,
    blocked: bool,
    alternatives: tuple[dict[str, Any], ...],
    angles: dict[str, float] | None,
    path: str | None,
    note: str | None,
)
    Outcome of ``move_to``.

    target_tool_centre / final_tool_centre: where the tool centre was sent, and where it
        ended.
    dist_to_target_m: distance between the two.
    pointing_error_deg: how far the final pointing direction is from the one asked (0
        when none was asked).
    rotation_error_deg: how far the whole final orientation is from the one asked — the
        pointing direction AND the fingers' turn about it.
    reached: within 0.015 m and 5 degrees of the whole orientation.
    steps_used: native actions consumed; ``steps_planned`` what the subdividing asked for
        (a motion stopped early because it arrived).
    truncated: it used every action ``max_actions`` allowed and had not arrived.
    stop_reason: why a motion that did not arrive stopped: ``"out_of_reach"`` (the arm
        kept moving but no longer got closer — the edge of its reach from this waist
        pose), ``"blocked"`` (the arm stopped moving: something is in the way, it pushes
        against what it holds — at the end of a push, the end of the travel — or a joint
        is at its end), ``"budget"`` (still closing in when the actions ran out: call
        again), or None when it arrived.
    blocked: ``stop_reason == "blocked"``.
    reach_hint: set with ``"out_of_reach"``.  Only then consider ``move_waist``.
    alternatives: when the target was refused before moving (``steps_used == 0`` and
        ``"out_of_reach"``: the benchmark's own IK, asked first, found no pose of this arm
        within 3 cm and 10 degrees of it), nearby requests that the same IK says ARE
        reachable — each a dict naming the change and the call(s) that make it, e.g.
        ``{"change": "pointing tilted 15 deg toward vertical", "move_to": {"xyz": [...],
        "point": [...]}}`` or one with ``"move_waist": {...}`` to run first.

class Gripper(arm: str, commanded: str, opening: float, holding: bool, steps_used: int)
    Outcome of ``set_gripper``.

    commanded: ``"closed"`` or ``"open"``.
    opening: the measured opening once the fingers stopped, normalised: 1.0 = open,
        0.0 = closed on nothing.
    holding: after a close, the fingers stopped apart: something is between them.  It
        does not say what: check that the intended object moves with the arm.
    steps_used: native actions consumed.

class Waist(
    moved: bool,
    reached: bool,
    chest_moved_m: tuple[float, float],
    torso_pitch_deg: float | None,
    turn_deg: float | None,
    error_reason: str | None,
    steps_used: int,
)
    Outcome of ``move_waist``.

    moved: the waist was commanded; False when the asked pose is outside the joints' range
        (``error_reason`` says which joint), and nothing moved.
    reached: the waist joints ended where they were sent.
    chest_moved_m: how far the chest went [forward, up], metres, in ``robo.FRAME``.  The
        hands moved by about the same amount: the arms keep their joint angles.
    torso_pitch_deg: how far the torso leans forward from upright (it starts near 28).
    turn_deg: how far the upper body is turned (+ to the robot's left).
    steps_used: native actions consumed.

class Home(arm: str, reached: bool, max_joint_error_rad: float, steps_used: int)
    Outcome of ``home``.

    reached: every joint of the returned arm(s) ended within 0.02 rad of the angle it had
        when the episode started.
    max_joint_error_rad: the largest difference that remains.
    steps_used: native actions consumed.

class ObjectPose(
    found: bool,
    centre_xyz: tuple[float, float, float] | None,
    top_z: float | None,
    height_m: float | None,
    size_xyz: tuple[float, float, float] | None,
    pixel_uv: tuple[int, int] | None,
    area_px: int,
    colour: str | None,
    camera: str,
    note: str | None,
    tilt_deg: float | None,
    long_axis_deg: float | None,
    support_z: float | None,
    method: str | None,
    score: float | None,
    candidates: tuple[dict[str, Any], ...],
    grasp_fingers: tuple[float, float, float] | None,
    grasp_yaw: float | None,
    width_m: float | None,
    length_m: float | None,
)
    Outcome of ``locate``: one colour region of the scene, measured in 3D.

    found: a region of contiguous colour was found in the head camera.
    centre_xyz: [x, y, z] of the middle of its footprint, at half its height above the
        surface it stands on, metres.
    top_z: the highest coordinate seen along ``robo.FRAME`` z.
    height_m: ``top_z`` above that surface.
    size_xyz: the region's extent along each axis of ``robo.FRAME``.
    pixel_uv: (row, col) of the region in the image it was measured in.
    area_px: pixels of the region.
    colour: mean colour of the region as a hex string, so a program can say which.
    camera: the camera measured from.
    note: why there is no 3D measurement, else None.
    tilt_deg: how far the object's upward face is tilted from horizontal, degrees (0 = lying
        flat / standing upright).  Tasks that check "upright" usually allow 5-15 degrees:
        re-``locate`` after placing and fix it if this is larger.
    long_axis_deg: direction of the long side of that face on the table, degrees from
        ``robo.FRAME`` +x (-90..90).  To close across the SHORT side, pointing down, turn the
        fingers along this line.
    support_z: height of the surface it stands on (the table top), metres.
    method: how the object was cut out of the image: ``"sam3_text"`` (SAM3 found the text
        you named), ``"geometry"`` (what rises above its surface at the pixel),
        ``"sam3_point"`` (SAM3's mask at the pixel), ``"colour"`` (the pixel's colour region).
    score: SAM3's confidence in its mask (0..1), when SAM3 made it.
    grasp_fingers: for an elongated object (a bottle, a pen, a box lying down), the
        horizontal direction ACROSS it in ``robo.FRAME``: the fingers close across the short
        side when they close along it (``grasp_yaw`` is the yaw that does so).  None for a
        round or square one (any direction closes on it).
    grasp_yaw: the ``yaw`` that, pointing straight down (``pitch=90``), closes the fingers
        across the object's narrow side: ``move_to([x, y, z], arm, pitch=90,
        yaw=pose.grasp_yaw)``.  None for a round or square object.
    width_m / length_m: the object's horizontal extent across and along its long axis —
        ``width_m`` is what the fingers must span.
    candidates: for a named object, every instance SAM3 matched to the name, best first:
        ``{"pixel_uv": (row, col), "score", "area_px"}``.  The measured one is the best
        scoring unless ``near`` chose another; when several match, check which is meant.

class Segment(
    found: bool,
    centroid: tuple[int, int] | None,
    area_px: int,
    colour: str | None,
    box: tuple[int, int, int, int] | None,
    camera: str,
    method: str | None,
    score: float | None,
)
    Outcome of ``segment``: a colour region of one camera.

    found: a region of contiguous colour was found.
    centroid: (row, col) of the region's centroid; pass it to ``back_project``.
    area_px: pixels of the region.
    colour: mean colour of the region as a hex string.
    box: (row0, col0, row1, col1) around the region.

class WorldPoint(
    pixel: tuple[int, int],
    xyz: tuple[float, float, float],
    depth_m: float,
    camera: str,
)
    Outcome of ``back_project``: the surface point seen at one pixel.

    xyz: [x, y, z] in ``robo.FRAME``, metres.  It is a visible SURFACE point, not a
        centre.
    depth_m: distance of that point along the camera's optical axis.

class BothMove(
    reached: bool,
    left: dict[str, Any] | None,
    right: dict[str, Any] | None,
    steps_used: int,
)
    Outcome of ``move_both``: ``left`` / ``right`` are each arm's ``final_tool_centre``,
    ``dist_to_target_m``, ``reached`` and ``stop_reason`` (None for an arm that held).

class BaseMove(commanded: dict[str, float], steps_used: int)
    Outcome of ``move_base``.  OPEN LOOP: ``commanded`` is what was sent, not a measured
    displacement; look with the head camera to see where the base went.

Exceptions:
    ToolError(RuntimeError): a call was rejected or a service failed; catch it if you have a fallback.
    ValueError: an argument was invalid (e.g. a planar move over 0.30 m) - nothing was executed.
    EpisodeFinished(BaseException): the episode is over (reason: truncated | timeout | cancelled); return.