RoboDojo

RoboDojo is a bimanual manipulation benchmark in Isaac Sim with a public leaderboard. PyRUA-Lean runs on it as an XPolicyLab policy: RoboDojo's own evaluation client runs the simulator and judges every episode, and PyRUA-Lean answers it as a policy server. A language model writes Python against robo, one cell at a time, while the episode runs. No VLA policy, no privileged information.

Overview

Robot	Two ARX-X5 arms with one gripper each; joint actions (env_cfg_type arx_x5, action_type joint)
Agent runtime	openai: any OpenAI-compatible chat endpoint (GPT-6 Astra in our evaluation)
Models on the policy side	SAM3 (segmentation); no VLA
Tasks	42 simulation tasks in five dimensions: Generalization, Precision, Long-Horizon, Memory, Open
Hardware	Linux, NVIDIA GPU; SAM3 takes about 8 GB on the policy side, Isaac Sim needs its own share

Observation and Action

Item	Description
Observation	RoboDojo's default observation: three colour images (cam_head, cam_left_wrist, cam_right_wrist), the joint and end-effector state of both arms, and the instruction. No depth image, no camera matrices, no object poses.
Action	The agent's cells call robo's primitives; each motion is solved to joint targets in the policy server and sent to the client as native actions.
Success	RoboDojo's own check of the scene. The agent never sees it; when a program finishes before the task is solved, RoboDojo judges the scene as it stands.
Task prompt	The task's instruction, and one knowledge text for every task: how the tools behave and how RoboDojo judges an episode (for most tasks: both arms back at their start pose with the grippers open). No task recipes.

Installation and Resources

1. Install RoboDojo with its XPolicyLab submodule and the simulator's environment, as RoboDojo describes.
2. Put the adapter into XPolicyLab: copy integrations/xpolicylab/PyRUA_Lean of this repository to XPolicyLab/policy/PyRUA_Lean (or link it from a PyRUA-Lean checkout).
3. In the policy environment, a conda environment (XPolicyLab's scripts conda activate it by name) with a CUDA build of torch and the torchvision built for it:

cd XPolicyLab/policy/PyRUA_Lean
bash install.sh              # XPolicyLab's server, PyRUA-Lean, the sam3 package
hf auth login                # facebook/sam3 is gated: accept its licence on Hugging Face first
bash download_assets.sh      # SAM3's checkpoint (3.4 GB) into ./checkpoints/sam3.pt

install.sh installs PyRUA-Lean from the checkout the directory is linked from, or from the public repository when the directory was copied; it keeps the environment's torch and torchvision as they are. download_assets.sh modelscope fetches the checkpoint from the ModelScope mirror, and SAM3_CHECKPOINT_PATH names a checkpoint kept elsewhere. The simulator's environment needs nothing beyond RoboDojo's own installation.

Model Configuration

The cells are written by a model behind an OpenAI-compatible chat endpoint:

export OPENAI_BASE_URL=https://YOUR_API_HOST/v1
export OPENAI_API_KEY=...
export PYRUALEAN_MODEL=gpt-6-astra          # the model's name at that endpoint

The adapter's deploy.yml sets the episode:

Key	Default	Meaning
max_turns	100	decision budget: code cells per episode
budget_s	5400	wall-clock limit of an episode, in seconds
images	on-demand	the model sees a camera only when its code asks for it (robo.show)
feedback	pure	a tool call returns what the code printed, nothing else
openai_options	max_tokens: 8192, nudges: 20, extra_body: {reasoning_effort: high}	the longest reply, the replies without a tool call tolerated, and JSON merged into every request

Task Selection

Option	Meaning
--task / task_name	A RoboDojo task, e.g. stack_blocks.
--seed / seed	The layout set: 0, 1 or 2, each with its own layouts of every task.
--eval-num	Episodes of the task; native runs RoboDojo's own number (50, or 25 for each half of a Generalization task). Start with 1: an episode took about 25 LLM calls and 13 minutes in the paper's runs, about $2.4 at GPT-6 Astra's gateway price.
--ckpt / ckpt_name	Ignored: there are no weights. no-checkpoint is the placeholder.

Run a Task

RoboDojo CLIAdapterWiring check

From the RoboDojo checkout, the official entry point:

bash scripts/robodojo.sh eval --policy-dir XPolicyLab/policy/PyRUA_Lean --task stack_blocks \
    --ckpt no-checkpoint --policy-env <policy env> --eval-env <simulator env> \
    --env-cfg arx_x5 --action-type joint --seed 0 --eval-num native

robodojo.sh server (on the policy machine) and robodojo.sh client (on the simulator machine, with --policy-host and --policy-port) run the two sides apart; give both --action-type joint and the same --ckpt, since their default is ee.

cd XPolicyLab/policy/PyRUA_Lean
bash eval.sh <bench_name> <task_name> <ckpt_name> <env_cfg_type> <action_type> <seed> \
  <policy_gpu_id> <env_gpu_id> <policy_conda_env> <eval_env_conda_env>

# example
bash eval.sh RoboDojo stack_blocks no-checkpoint arx_x5 joint 0 0 0 pyrualean RoboDojo

No simulator and no model call: XPolicyLab's debug client exercises the loop, the action keys and their dimensions.

cd XPolicyLab/policy/PyRUA_Lean
EVAL_ENV_TYPE=debug bash eval.sh RoboDojo stack_bowls no-checkpoint arx_x5 joint 0 0 0 pyrualean RoboDojo

View Results

• RoboDojo records every episode under eval_result/ of its checkout; bash scripts/robodojo.sh summarize writes the leaderboard's table (SR and Score per dimension) to eval_result/RoboDojo/_summary.md. A task counts there only once all its episodes for that seed are in.
• Each episode's programs, tool calls and full transcript are kept under the adapter's result_dir: ./results/<task>_l<layout>_s<seed>-<time>/.

Results

GPT-6 Astra, seed 0, all 42 tasks (2,100 episodes):

	Generalization	Precision	Long-Horizon	Memory	Open	Average
SR (%)	45.83	37.25	48.00	63.00	64.00	51.62
Score	50.65	47.02	63.26	63.00	67.09	58.20

That is 19.4 points above the best entry (32.26) of a snapshot of the public leaderboard taken on October 9, 2026; the board changes as entries arrive. An episode takes 25.7 LLM calls on average.

What Runs Where

• Evaluation client (RoboDojo, in the simulator's environment) owns Isaac Sim: it builds the observation, counts the task's action budget and gives the verdict.
• Policy server (XPolicyLab's server with the adapter, in the policy environment) runs the PyRUA-Lean episode: the agent loop, the sandboxed cells, the primitives (inverse kinematics, gripper, perception) and SAM3.
• Model endpoint writes the cells.

The two sides talk XPolicyLab's websocket protocol (update_obs / get_action). A model may think for minutes between two actions, so the policy server answers "wait" until the next action is ready and the client asks again; no request is held longer than poll_s (20 s).

Primitives the Agent Can Call

Motion:

• move_to(xyz, arm, gripper, pitch, yaw, roll, path): move one arm's grasp point to a world position, with an orientation. A pose out of reach moves nothing, and the result names reachable poses nearby.
• set_gripper(arm, gripper, opening=None): close or open one gripper until the fingers stop; the result says whether something is held and how wide it is.
• home(arm="both"): return an arm to the joint angles it had when the episode started.

Perception and state:

• locate(text_prompt, camera): find an object by words and measure it in 3D from the colour cameras: centre, top, footprint, heading and the yaw of a top grasp across it.
• segment(text_prompt, camera): the SAM3 mask of an object in the current image.
• back_project(row, col, camera): the world point seen at one pixel.
• state(), image(camera), show(camera), camera_meta(camera), artifact(name): the recorded state of both arms, the images, and what the model looks at.

Twists and button presses are compositions of these; the knowledge text says how.

API reference: what the agent reads, generated from pyrualean.robodojo_xpl_rgb when these docs are built

class RobodojoOfficialColourRobot
    The RoboDojo dual-arm station: two 6-DoF ARX-X5 arms with parallel grippers at a table.

    World frame: x runs across the table (the LEFT arm is at negative x and
    appears on the LEFT of the head image), y points away from the robot
    towards the far edge of the table, z is up; metres.  Orientations are unit
    quaternions ordered [w, x, y, z].  The arms are ``"left"`` and ``"right"``
    (``robo.LEFT`` / ``robo.RIGHT``); a motion call names one arm and holds
    the other where it is.

    ``move_to`` drives an arm's GRASP POINT, the middle of the finger pads
    (``state().left.tcp_pos``), to a world position, with an orientation given
    in degrees: ``pitch`` is how far the gripper points below the horizontal
    (0 level, 90 straight down), ``yaw`` the heading of the pointing direction
    on the table (0 away from the robot, +90 towards the robot's left, -90
    towards its right) and ``roll`` the turn of the fingers about the pointing
    direction (0: side by side, as at the start).  Pointing straight down, the
    fingers close along the line at ``yaw`` degrees from +x (counter-clockwise
    seen from above): ``locate``'s ``grasp_yaw`` is that line across an
    object's narrow side.  The table
    surface is at z = 0.765; the arms stand on it at x = -0.30 and +0.30,
    y = -0.45.

    Two gripper conventions, do not mix them up.  COMMANDS are signed:
    ``robo.CLOSE`` (1) closes, ``robo.OPEN`` (-1) opens, ``robo.KEEP`` (0)
    leaves the gripper as it is.  MEASUREMENTS (``state().left.gripper``,
    ``Gripper.opening``) are openings: 1.0 = open, 0.0 = closed on nothing.

    Every native action (one step of ``move_to``, ``set_gripper`` or
    ``home``) counts against the episode's
    native-action budget; when it is spent the state reports ``truncated``
    and motion calls raise ``EpisodeFinished``.  Motion calls block until the
    primitive finishes and return a typed result; nothing is queued.

    Perception is from colour images alone; there is no depth image.  A fixed
    head camera (``cam_head``, the whole table) and one wrist camera per arm
    (``cam_left_wrist``, ``cam_right_wrist``), 640x480 each, and where each
    one is, is known from the robot's state.  ``locate`` finds an object by a
    text prompt and measures it in 3D (centre, top, footprint, heading) from
    the cameras that see it: it takes two of them to measure a height.
    ``segment`` / ``back_project`` give a mask centroid and the world point at
    one pixel.  Object poses are never given.

Constants:
    robo.ARMS = ('left', 'right')
    robo.LEFT = 'left'
    robo.RIGHT = 'right'
    robo.CLOSE = 1
    robo.OPEN = -1
    robo.KEEP = 0

Methods and attributes of `robo`:

robo.task -> str  (read-only attribute)
    The task instruction the benchmark generated for this layout (authoritative).

    Read it before planning.

robo.state(step: int = -1) -> State
    Read the recorded state of both arms.

    Args:
        step: recorded step to read; ``-1`` is the latest, ``0`` the
            initial scene.  A new step is recorded after every motion call.
    Returns:
        State with ``left`` / ``right`` end-effector poses, measured
        gripper openings and joint angles, and the ``truncated`` flag.

robo.image(camera: str = 'cam_head', *, resolution: str = 'high', step: int = -1) -> Any
    Load a recorded camera image as a ``numpy`` uint8 array (480, 640, 3), RGB.

    Args:
        camera: ``"cam_head"`` (fixed view of the whole table: identify
            objects, destinations and the global layout),
            ``"cam_left_wrist"`` or ``"cam_right_wrist"`` (on the gripper:
            what that arm is about to grasp or is holding).
        resolution: there is one resolution per camera, ``"high"``.  Pixel
            coordinates are (row, col) with row 0 at the top and col 0 at
            the left.
        step: recorded step, ``-1`` = latest.  ``segment`` and
            ``back_project`` always read the CURRENT observation, which is
            the latest recorded step.

robo.show(camera: str = 'cam_head') -> None
    Attach the current image of ``camera`` to the feedback of this turn.

    Images are not returned automatically; call ``show("cam_head")``,
    ``show("cam_left_wrist")`` or ``show("cam_right_wrist")`` when *you*
    (the author) need to look at the scene before writing the next turn.
    Costs tokens: look only when a decision depends on it.

robo.artifact(name: str, *, step: int = -1) -> bytes
    Return the raw bytes of a recorded artifact (advanced).

    Names are ``cam_head.png``, ``cam_left_wrist.png`` and
    ``cam_right_wrist.png``.

robo.camera_meta(camera: str = 'cam_head', *, step: int = -1) -> dict[str, Any]
    Return the calibration of one camera at a recorded step as a dict.

    Keys: ``shape`` (image shape), ``intrinsic_matrix`` (3x3) and
    ``extrinsic_matrix`` (4x4 camera-to-world; the camera looks down its
    own -z axis with +y up).  The head camera is fixed; a wrist camera
    moves with its arm, and its pose is worked out from that arm's state.

robo.segment(
    text_prompt: str,
    camera: str = 'cam_head',
    min_score: float = 0.2,
) -> Segment
    Segment an object in the current image of ``camera`` by a text prompt (SAM3).

    Args:
        text_prompt: what to find, e.g. ``"the green bottle"``.  Name one
            object by its visible properties.
        camera: ``"cam_head"`` (default), ``"cam_left_wrist"`` or
            ``"cam_right_wrist"``.
        min_score: masks scoring below this are reported as not found.
    Returns:
        Segment; when ``found``, pass ``centroid`` (row, col) to
        ``back_project`` with the same camera to get a world point.  The
        mask is the best match for the text, not a guarantee of identity:
        check where it is against the image.
    Raises:
        ToolError: the camera image or the SAM3 service is unavailable.

robo.back_project(row: int, col: int, camera: str = 'cam_head') -> SeenPoint
    Return the world point seen at one pixel of the current image of ``camera``.

    There is no depth image, so a pixel alone is only a ray.  The object
    under the pixel is segmented from that pixel (SAM3) and measured as
    ``locate`` measures one, and the answer is where the pixel's ray
    meets it.  A pixel on the table top itself gives the point of the
    table it shows, exactly.

    Args:
        row / col: pixel, row 0 at the top, col 0 at the left (images are
            480 rows x 640 cols).
        camera: the camera the pixel was read from (default ``"cam_head"``).
    Returns:
        SeenPoint; check ``surface``.  ``"object"``: ``xyz`` is on the
        object under the pixel, as well as ``locate`` would measure that
        object from this camera and the others that see it.
        ``"table"``: ``xyz`` is on the table top; a pixel that shows
        something too flat or too far to measure also answers so.  A thin
        part in mid-air (a peg, a handle, a cable) is not measured
        reliably: reason from the image and from ``locate`` of the whole
        object instead.
    Raises:
        ToolError: pixel out of range, or it looks past the table.

robo.locate(
    text_prompt: str,
    camera: str = 'cam_head',
    min_score: float = 0.2,
) -> ObjectSight
    Find an object by a text prompt and measure it in 3D from the colour cameras.

    SAM3 segments the current image of ``camera`` (as ``segment``); the
    same object is then looked for in the other two cameras, and its
    height and footprint are those that fit every camera that shows it.
    One camera cannot measure a height: a low deep thing and a tall
    shallow one look alike from it.  ``views`` says how many cameras
    agreed.  Nothing moves.

    Args:
        text_prompt: one object by its visible properties, e.g. ``"the
            small red cube on the left"``.
        camera: ``"cam_head"`` (default; the whole table),
            ``"cam_left_wrist"`` or ``"cam_right_wrist"`` (close up, from
            the gripper).
        min_score: masks scoring below this are reported as not found.
    Returns:
        ObjectSight.  For a top grasp: ``move_to([x, y, top_z + 0.09],
        pitch=90)`` above it, ``locate`` it again from there (the wrist
        camera then looks down on it and the reading is good to 1-2 mm),
        turn to ``yaw=grasp_yaw`` and go down to the height to hold it
        at.  With ``views >= 2`` and no ``note`` a block, a can, a bowl
        or the body of a bottle is measured to 1-3 mm; a ``note`` that
        the cameras agree only roughly leaves the height uncertain.  With
        ``views == 1`` the height is a guess, the tallest upright shape
        that fits the one image: close for a box or a can, wrong for a
        bowl, a cup or anything wider above than below, and for a tall
        thing that tapers; ``center_xyz`` is then off, along the line
        away from the robot, by about half of what the height is wrong
        by.  An object once measured with ``views >= 2`` keeps that height
        in later calls with the same prompt.  No position comes back
        (``center_xyz`` None, ``note`` says why) for a mask on the robot's
        own arm or on something that is not on the table.  The mask is
        the best match for the text, not proof of identity: check it
        against the image.
    Raises:
        ToolError: the camera image or the SAM3 service is unavailable.

robo.move_to(
    xyz: Sequence[float],
    arm: str = 'right',
    gripper: int = 0,
    pitch: float | None = None,
    yaw: float | None = None,
    roll: float | None = None,
    path: str = 'auto',
) -> Move
    Move one arm's grasp point to a world position, with an orientation.

    The grasp point is the middle of the finger pads (``tcp_pos``).  The
    library solves the arm's inverse kinematics for the full pose and,
    with ``path="auto"``, moves the grasp point along a straight line to
    it, about 2.5 cm or 8 degrees per native action; when the line leaves
    the arm's reach it swings the joints straight to the solution instead.
    Nothing moves when the pose is out of reach or every way there takes
    the arm into the table: ``ik_error`` says why.

    Args:
        xyz: target grasp-point position [x, y, z], metres.
        arm: ``"left"`` or ``"right"``.  The other arm holds its joints.
        gripper: ``robo.CLOSE`` (1) or ``robo.OPEN`` (-1) to command the
            moving arm's gripper on every action of the motion,
            ``robo.KEEP`` (0) to keep commanding what it was last told (a
            held object stays squeezed).  Keep it closed (1) while
            carrying an object.
        pitch / yaw / roll: the orientation to arrive with, degrees (the
            class docstring defines them).  An angle left None keeps its
            current value, so a plain position move does not turn the
            gripper.  With an EMPTY open gripper the fingers are
            interchangeable, so the pose turned half a turn about the
            pointing direction may be taken when it needs less joint
            travel (``note`` says so; yaw / roll then read 180 off);
            holding something, the pose is taken exactly as asked.
        path: ``"auto"`` (default), ``"line"`` (straight line or nothing)
            or ``"joint"`` (swing the joints: fewer actions, but the grasp
            point does not follow a line: not near objects).
    Returns:
        Move; test ``reached`` (within 0.01 m and 5 degrees).  ``blocked``
        means the arm was stopped by something on the way.
    Raises:
        ValueError: a malformed argument (nothing is sent).

robo.set_gripper(arm: str, gripper: int, opening: float | None = None) -> Gripper
    Close or open one gripper, without moving the arms.

    The command is repeated until the fingers stop (2 to 10 native
    actions).  After a close, ``holding`` tells whether something stopped
    the fingers apart and ``width_m`` how wide it is.  Later motions with
    ``robo.KEEP`` keep commanding what was commanded last, so a held
    object stays squeezed and a partial opening stays as it is.

    Args:
        arm: ``"left"`` or ``"right"``.
        gripper: ``robo.CLOSE`` (1) or ``robo.OPEN`` (-1).
        opening: with ``robo.OPEN`` only: open to this reading
            (0 < opening <= 1) instead of all the way, e.g. to reach
            between close neighbours.  Something 0.002 + 0.109 x opening
            metres wide then just fits between the fingers.
    Returns:
        Gripper.  ``holding`` is not proof that the INTENDED object is
        held: confirm it moves with the arm (wrist camera, or ``locate``
        it again after a lift).

robo.home(arm: str = 'both') -> Home
    Return an arm to the joint angles it had when the episode started.

    A grasp point lower than 1.0 m first goes straight up by up to 6 cm;
    then each native action commands the starting angles directly, so the
    arm swings straight back: no IK and no path planning beyond that lift
    (clear tall things yourself).  The call returns when every joint is
    within 0.01 rad or after 25 actions of the swing.  The gripper keeps
    the opening it has; an arm that is not named holds its joints.  Most
    tasks count as done only with both arms back where they started.

    Args:
        arm: ``"left"``, ``"right"`` or ``"both"`` (default).
    Returns:
        Home; test ``reached`` and call again when it is False.

Result types (frozen dataclasses; access fields as attributes):

class State(step: int, task: str, left: ArmState, right: ArmState, truncated: bool)
    Snapshot of both arms at one recorded step.

    step: index of the recorded step (0 = initial scene).  A new step is
        recorded after every motion call.
    task: the task instruction given by the benchmark (authoritative).
    left / right: the two arms (``state.arm("left")`` picks one by name).
    truncated: the native-action budget of the episode is exhausted; no
        further motion is possible.

class ArmState(
    tcp_pos: tuple[float, float, float],
    pitch: float,
    yaw: float,
    roll: float,
    eef_pos: tuple[float, float, float],
    eef_quat: tuple[float, float, float, float],
    gripper: float,
    joints: tuple[float, ...],
)
    One arm at a recorded step.

    tcp_pos: the grasp point [x, y, z] in the world frame, metres: the middle of
        the finger pads, 0.145 m out along the pointing direction from the
        end-effector link.  ``move_to`` drives this point.
    pitch / yaw / roll: the gripper's orientation, degrees (see the class
        docstring): ``pitch`` 0 points level and 90 straight down, ``yaw`` is
        the heading of the pointing direction on the table (0 away from the
        robot, +90 towards the robot's left), ``roll`` the turn of the fingers
        about the pointing direction (0 as at the start).  Pointing straight
        down, ``yaw`` alone says how the fingers lie and ``roll`` reads 0.
    eef_pos: position of the end-effector link (the wrist flange), metres.
    eef_quat: end-effector orientation as a unit quaternion [w, x, y, z].
    gripper: measured gripper opening, normalised: 1.0 = open, 0.0 = closed on
        nothing; closed on an object 0.002 + 0.109 x this reading is the
        object's width in metres.
        (Gripper COMMANDS use the opposite sign convention: see ``set_gripper``.)
    joints: the six arm joint angles, radians (the last turns the fingers about
        the pointing direction).

class Move(
    arm: str,
    target_xyz: tuple[float, float, float],
    start_xyz: tuple[float, float, float],
    final_xyz: tuple[float, float, float],
    dist_to_target_m: float,
    orientation_error_deg: float,
    pitch: float,
    yaw: float,
    roll: float,
    reached: bool,
    path: str | None,
    ik_error: str | None,
    note: str | None,
    blocked: bool,
    steps_used: int,
    truncated: bool,
)
    Outcome of ``move_to``.

    target_xyz: the requested grasp-point position.
    start_xyz / final_xyz: where the grasp point was before and after.
    dist_to_target_m: distance from ``final_xyz`` to ``target_xyz``.
    orientation_error_deg: how far the final orientation is from the one asked
        (the angles passed, the others as they were).
    pitch / yaw / roll: the orientation the gripper ended with, degrees.
    reached: within 0.01 m and 5 degrees of the target.
    path: ``"line"`` (the grasp point moved along a straight line),
        ``"joint"`` (the joints swung straight to the solution) or None
        (nothing moved).
    ik_error: why nothing moved (the pose is out of reach, or every way there
        goes through the table), or None.  An out-of-reach target names the
        nearby poses that ARE reachable (a less steep pitch here, or this
        orientation higher or lower).
    note: set when the exact pose was out of reach but one within 2 cm and
        8 degrees was not: the arm went there instead (``dist_to_target_m``
        and ``orientation_error_deg`` say how far off it is).
    blocked: the arm stopped short of a planned pose it could reach: something
        is in the way, or a held object is pushing against something.
    steps_used: native actions consumed.
    truncated: the native-action budget ran out.

class Gripper(
    arm: str,
    commanded: str,
    opening: float,
    holding: bool,
    width_m: float | None,
    steps_used: int,
    step: int | None,
    step_limit: int | None,
    truncated: bool,
)
    Outcome of ``set_gripper``.

    commanded: ``"closed"`` or ``"open"``.
    opening: the opening once the fingers stopped (1.0 = open, 0.0 = closed on
        nothing).
    holding: after a close, the fingers stopped apart (an opening above 0.01,
        about 3 mm): something is between them.  It does not say what: check
        that the intended object moves with the arm.
    width_m: after a close, the width of what is held, across the fingers
        (0.0 when nothing is), else None.
    steps_used: native actions consumed.
    step / step_limit: native actions executed so far and the episode budget
        (None if the runtime did not report them).
    truncated: the native-action budget ran out.

class Home(
    arm: str,
    reached: bool,
    max_joint_error_rad: float,
    steps_used: int,
    truncated: bool,
)
    Outcome of ``home``.

    arm: ``"left"``, ``"right"`` or ``"both"``, as requested.
    reached: every joint of the returned arm(s) ended within 0.01 rad of the
        angle it had when the episode started.
    max_joint_error_rad: the largest difference that remains.
    steps_used: native actions consumed.
    truncated: the native-action budget ran out.

class ObjectSight(
    found: bool,
    center_xyz: tuple[float, float, float] | None,
    top_z: float | None,
    height_m: float | None,
    length_m: float | None,
    width_m: float | None,
    long_axis_deg: float | None,
    grasp_yaw: float | None,
    score: float | None,
    centroid: tuple[int, int] | None,
    area_px: int,
    camera: str,
    note: str | None,
    views: int,
)
    Outcome of ``locate``: an object measured from its masks in the colour cameras.

    found: SAM3 produced a mask scoring at least ``min_score``.
    center_xyz: [x, y, z] of the middle of the smallest rectangle around the
        object's footprint, at half its height above the table, metres.  A
        handle, a spout or a tail is part of the footprint and moves the
        middle away from the body's.  None when no 3D measurement was
        possible (see ``note``).
    top_z: height of the object's top surface.
    height_m: ``top_z`` above the table surface (0.765 m).
    length_m / width_m: the long and the short side of the smallest rectangle
        around the footprint.
    long_axis_deg: direction of the long side on the table, degrees from +x
        (-90 .. 90); of a square, one of its sides.
    grasp_yaw: the ``yaw`` for ``move_to(..., pitch=90, yaw=grasp_yaw)``:
        pointing straight down, the fingers then close across the short side
        (a cube's: across two faces).
    views: how many cameras agreed on the measurement.  1: nothing but the
        camera asked saw the object, and its height is a guess (see
        ``locate``); ``note`` says which guess.
    score: SAM3 confidence.
    centroid: (row, col) of the mask in the image, as ``Segment.centroid``.
    area_px: mask pixels.
    camera: the camera measured from.
    note: what limits the measurement (one camera only; two that agree only
        roughly; the mask is the robot's own arm, or not on the table; too
        flat or too far to measure), else None.

    The object is taken to stand on the table.  What other things or the
    robot's own arm hide of it is not measured, so a partly hidden object
    comes out smaller and off-centre.  A held object is usually not found from
    the head camera.

class Segment(
    found: bool,
    centroid: tuple[int, int] | None,
    area_px: int,
    score: float | None,
    box: Any,
    camera: str,
)
    Outcome of ``segment``.

    found: SAM3 produced a mask scoring at least ``min_score``.
    centroid: (row, col) of the mask centroid (mean of the mask pixels, floored),
        or None.  Pass exactly this pixel, with the same camera, to
        ``back_project``.
    area_px: number of mask pixels (0 when not found).
    score: SAM3 confidence in [0, 1], None when not found.
    box: mask bounding box as reported by SAM3, or None.

class SeenPoint(
    pixel: tuple[int, int],
    xyz: tuple[float, float, float],
    depth_m: float,
    camera: str,
    surface: str,
)
    Outcome of ``back_project``: the point seen at one pixel.

    xyz: world [x, y, z], metres: where the pixel's ray meets the object under
        the pixel, or the table.
    depth_m: distance of that point along the camera's optical axis.
    surface: ``"object"`` when the pixel lies on something that could be
        measured, ``"table"`` when the point is on the table top (z = 0.765):
        the pixel shows the table, or something too flat or too far to measure.

Exceptions:
    ToolError(RuntimeError): a call was rejected or a service failed; catch it if you have a fallback.
    ValueError: an argument was invalid (e.g. a planar move over 0.30 m) - nothing was executed.
    EpisodeFinished(BaseException): the episode is over (reason: truncated | timeout | cancelled); return.