LIBERO-PRO

Runs on RPent

LIBERO-PRO runs on RPent's LIBERO stack: RPent boots the simulator, the Pi0.5 VLA and SAM3, and every robo call goes through RPent's toolkit. New robots are added through XPolicyLab (Adding a Robot).

LIBERO-PRO perturbs LIBERO's tabletop tasks: swapped objects and positions, rephrased or changed instructions. The robot is a simulated Franka arm with RPent's scripted servos, the frozen Pi0.5 VLA skills and SAM3 perception.

Overview

Robot	Franka arm in MuJoCo / robosuite
Action models	Pi0.5 (pi0_pick, pi0_doubled); SAM3 for segmentation
Agent runtimes	codex (default), claude, openai (Agent Configuration)
Tasks	The LIBERO-PRO suites RPent runs, e.g. libero_object_swap, libero_spatial_swap
Hardware	Linux, NVIDIA GPU, EGL; Python 3.11

Observation and Action

Item	Description
Observation	The agentview and wrist images, their per-pixel world coordinates, the end-effector pose, the gripper opening and the scene's object names.
Action	The agent's cells call robo; each call is one RPent tool (a scripted servo or a Pi0.5 skill) that steps the simulator.
Success	LIBERO's own task predicate. robo.done turns true once it fires, and every further motion is refused.
Task prompt	The suite's task instruction, a task card (the step-0 images and the initial gripper pose) and the robot's knowledge text.

Installation and Resources

RPent at commit 886b3b2, in its own Python 3.11 environment, with PyRUA-Lean in the same environment:

git clone https://github.com/RLinf/RPent.git && cd RPent && git checkout 886b3b2
uv venv --python 3.11 && source .venv/bin/activate
uv pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu128
uv pip install -e ".[libero-pro]"
uv pip install -e /path/to/PyRUA-Lean
liberopro-download-assets --assets-dir /path/to/liberopro-assets

liberopro looks for its assets inside its own package: replace site-packages/liberopro/liberopro/assets by a symlink to the assets directory. Then the environment:

export LIBERO_PRO_ASSET_PATH=/path/to/liberopro-assets
export LIBERO_TYPE=pro ROBOT_PLATFORM=LIBERO MUJOCO_GL=egl PYOPENGL_PLATFORM=egl XLA_PYTHON_CLIENT_PREALLOCATE=false
export LIBERO_CONFIG_PATH=/path/to/a/writable/dir
export OMP_NUM_THREADS=4 OPENBLAS_NUM_THREADS=4 MKL_NUM_THREADS=4      # as in the paper's runs

VLA Configuration

Pi0.5 RLinf/RLinf-Pi05-LIBERO-130-fullshot-SFT from Hugging Face (optimizer.pt is not needed):

hf download RLinf/RLinf-Pi05-LIBERO-130-fullshot-SFT --local-dir /path/to/pi05
export PI05_CHECKPOINT_PATH=/path/to/pi05

SAM3 Configuration

hf auth login                                                     # facebook/sam3 is gated: accept its licence first
hf download facebook/sam3 sam3.pt --local-dir /path/to/sam3
export SAM3_CHECKPOINT_PATH=/path/to/sam3/sam3.pt

Task Selection

Option	Meaning
--suite	A LIBERO-PRO suite, e.g. libero_object_swap.
--task	Zero-based task index within the suite.
--seed	Environment seed (default 0).
--libero-type	The installed LIBERO variant (default pro).

Run a Task

export RPENT_ROOT=/path/to/rpent
pyrualean play --arm cells --suite libero_spatial_swap --task 7 --seed 2 \
    --model gpt-6-astra --max-turns 40 --budget-s 7200 --out runs/demo --cuda-device 0
pyrualean summarize runs/demo

An episode with private servers boots Pi0.5 and SAM3 (~10 GB of GPU memory, ~100 s); attached to running servers (--vla-endpoint / --sam3-endpoint) it needs ~1.5 GB and ~15 s. --guides gives the agent RPent's three LIBERO operating guides (Running Episodes).

View Results

The run directory keeps the episode video (episode.mp4), every cell the agent ran (workspace/) and what each printed (the runtime's event file, e.g. codex_events.jsonl), the ledger of primitive calls (calls.jsonl), the token usage and LIBERO's verdict (result.json), and RPent's own records (states.json, per-step images).

Results

GPT-6 Astra, 40 tasks at five seeds (200 task instances), at most 40 LLM calls and two hours per episode. Against RPent's tool-calling agent with the same planner and primitives; LLM calls, input tokens and cost per solved episode on the instances both solved:

	Success (%)	LLM calls	Input tokens	Cost
Tool calling (RPent)	83.0	21.4	1.23M	$2.49
PyRUA-Lean	94.0	8.4	273k	$0.79

What Runs Where

• Env server (RPent) owns the LIBERO MuJoCo environment and its EGL rendering.
• Pi0.5 server and SAM3 server (RPent) hold the VLA and the segmentation model.
• PyRUA-Lean boots them for the episode, builds RPent's toolkit and serves the agent's cells against robo.
• Agent runtime (Codex CLI, Claude Code or the openai loop) talks to the model.

Primitives the Agent Can Call

Motion:

• move_to(xyz, gripper, ...): servo the gripper to a world position, holding its orientation.
• move_pose(xyz, target_pitch=..., target_yaw=..., ...): servo position and wrist orientation together.
• rotate_wrist(target_yaw= / delta_yaw=), rotate_pitch(target_pitch= / delta_pitch=): turn or tilt the gripper.
• set_gripper(gripper, steps), release(): drive the gripper; open it and let go.
• pi0_pick(prompt): let the frozen Pi0.5 VLA perform a closed-loop grasp.
• pi0_doubled(prompt): let Pi0.5 perform a contact skill that is not a pick.

Perception and state:

• segment(prompt | point=...): locate an object with SAM3 and return its world position.
• back_project(row, col), region_center(row_range, col_range): the world point under a pixel, the centre of a pixel window.
• state(), image(camera), world_map(camera), show(camera), camera_meta(camera), artifact(name), done.

How It Works gives the rules every call follows.

API reference: what the agent reads, generated from pyrualean.libero when these docs are built

class LiberoRobot
    A LIBERO tabletop Franka arm with a two-finger gripper.

    World frame: x/y span the table, z is up, units are metres and radians.
    The arm starts at a home pose above the table, gripper open and pointing
    down.  Motion calls block until the primitive finishes and return a typed
    result describing what actually happened; nothing is queued.

    Gripper commands are floats: ``LiberoRobot.OPEN`` (-1) opens,
    ``LiberoRobot.CLOSE`` (+1) closes and keeps holding.  Every motion call
    takes a ``gripper`` argument that is *held for the whole motion*; carrying
    an object with the default ``gripper=-1`` silently drops it.

    Perception is camera based: the scene is observed through a fixed
    ``agentview`` camera (global layout) and a ``wrist`` camera (close range).
    Object poses are never given; use ``segment`` / ``back_project`` on the
    recorded images to obtain world coordinates.

Constants:
    robo.OPEN = -1.0
    robo.CLOSE = 1.0

Methods and attributes of `robo`:

robo.task -> str  (read-only attribute)
    The task instruction, e.g. ``"put the bowl on the plate"``.

robo.done -> bool  (read-only attribute)
    True once the benchmark's success predicate has fired (sticky).

robo.state(step: int = -1) -> State
    Read the recorded proprioceptive state.

    Args:
        step: recorded step to read; ``-1`` is the latest, ``0`` the
            initial scene.  A new step is recorded after every motion call.
    Returns:
        State with gripper position, orientation, finger opening, the
        object names and the episode flags.

robo.image(
    camera: str = 'agentview',
    *,
    resolution: str = 'high',
    step: int = -1,
) -> Any
    Load a recorded camera image as a ``numpy`` uint8 array (H, W, 3), RGB.

    Args:
        camera: ``"agentview"`` (fixed global view, best for identifying
            objects and layout) or ``"wrist"`` (camera on the gripper,
            best for close-range geometry).
        resolution: ``"high"`` (1024x1024) or ``"low"`` (256x256).  Pixel
            coordinates are (row, col) with row 0 at the top and col 0 at
            the left; pass the same resolution to ``back_project``.
        step: recorded step, ``-1`` = latest.

robo.world_map(
    camera: str = 'agentview',
    *,
    resolution: str = 'high',
    step: int = -1,
) -> Any
    Load the per-pixel world coordinates of a recorded image.

    Returns a ``numpy`` float array (H, W, 3): ``world_map[row, col]`` is
    the [x, y, z] of the surface seen at that pixel (zeros / non-finite
    where depth is invalid).  This is what ``back_project`` reads; load it
    when you want to reason about many pixels at once.

robo.show(camera: str = 'agentview') -> None
    Attach the current image of ``camera`` to the feedback of this turn.

    Images are not returned automatically; call ``show("agentview")`` or
    ``show("wrist")`` when *you* (the author) need to look at the scene
    before writing the next turn.  Costs tokens: look only when a
    decision depends on it.

robo.artifact(name: str, *, step: int = -1) -> bytes
    Return the raw bytes of a recorded artifact (advanced).

    Names include ``agentview_high.png``, ``wrist_high.png``,
    ``agentview_depth.npz`` and the ``overlay_artifact`` of a ``segment``.

robo.camera_meta(camera: str = 'agentview', *, step: int = -1) -> dict[str, Any]
    Return the camera calibration metadata (intrinsics/extrinsics) as a dict.

robo.segment(
    prompt: str | None = None,
    *,
    camera: str = 'agentview',
    point: tuple[int, int] | None = None,
    min_score: float = 0.2,
    step: int = -1,
) -> Segment
    Locate an object with SAM3 on a recorded image and return its world position.

    Give exactly one of ``prompt`` (a short visual description) or
    ``point`` (a positive pixel ``(row, col)`` in the high-resolution
    image).  The top-ranked mask is projected through the world map, so
    ``world_xyz`` is the object's visible *surface* median: use its x/y for
    a target and pick z from the object's resting height.

    Args:
        prompt: colour + shape + relation, e.g. ``"the black bowl on the
            stove"``.  Brand or internal names (``"akita"``,
            ``"alphabet soup"``) score badly; describe what it looks like.
        camera: ``"agentview"`` for global identity, ``"wrist"`` for
            refinement once the gripper is 15-20 cm above the target.
        point: alternative to ``prompt``; a pixel on the object.
        min_score: reject masks below this confidence.
        step: recorded step whose image to use (``-1`` = latest).
    Returns:
        Segment; check ``found`` before using ``world_xyz``.

robo.back_project(
    row: int,
    col: int,
    *,
    camera: str = 'agentview',
    resolution: str = 'high',
    step: int = -1,
) -> BackProjection
    Return the world [x, y, z] of the surface seen at pixel ``(row, col)``.

    The pixel must come from the same camera and resolution as requested
    (``image(camera, resolution=...)``).  Sample several pixels firmly on
    an object's top surface and take the median; pixels on thin rims,
    edges or the gap to the table hit the background metres away.

    Raises:
        ToolError: pixel out of bounds or no valid depth at that pixel.

robo.region_center(
    row_range: tuple[int, int],
    col_range: tuple[int, int],
    *,
    camera: str = 'agentview',
    resolution: str = 'high',
    z_min: float | None = None,
    z_max: float | None = None,
    step: int = -1,
) -> RegionCenter
    Return the world centre of a pixel window (e.g. a container cavity).

    Uses the midpoint of the window's world x/y extent rather than a
    median, which is biased towards a rim or edge.  ``z_min`` / ``z_max``
    keep only pixels inside a world-z band (e.g. the basket floor).

    Raises:
        ToolError: empty window or too few valid pixels.

robo.move_to(
    xyz: list[float],
    gripper: float = -1.0,
    *,
    tol: float = 0.012,
    step_clip: float = 0.025,
    max_steps: int = 80,
    target_yaw: float | None = None,
    yaw_step_clip: float = 0.1,
    action_scale: float = 0.05,
) -> Move
    Servo the gripper to a world-frame position, holding its orientation.

    Args:
        xyz: target [x, y, z] in metres.  One call may move at most
            0.30 m in the x/y plane; longer traversals raise ValueError -
            split them into 2-3 waypoints at carrying height.
        gripper: -1 open, +1 close; held during the whole motion.
        tol: stop when within this distance (m).
        step_clip: per-step travel cap (m): 0.025 empty/box, 0.015 cans,
            0.012 tall bottles or fine approaches.
        max_steps: simulator step budget.
        target_yaw: optional world yaw (rad) to servo simultaneously.
        yaw_step_clip: per-step yaw cap (rad) when ``target_yaw`` is set.
        action_scale: OSC action scale; leave at the default.
    Returns:
        Move; check ``reached`` - a stalled servo returns normally.

robo.move_pose(
    xyz: list[float],
    *,
    target_pitch: float | None = None,
    target_yaw: float | None = None,
    gripper: float = -1.0,
    step_clip: float = 0.02,
    pitch_step: float = 0.08,
    yaw_step: float = 0.08,
    tol: float = 0.012,
    ori_tol: float = 0.05,
    max_steps: int = 150,
    action_scale: float = 0.05,
) -> MovePose
    Servo position and wrist orientation (pitch/yaw) at the same time.

    Use it where ``move_to`` stalls short of a low or recessed target
    (cabinet fronts, microwave cavities): co-varying tilt and position
    threads the IK singularity that a fixed gripper-down servo hits.
    ``gripper`` defaults to open; pass ``+1`` when holding an object.

robo.rotate_wrist(
    *,
    target_yaw: float | None = None,
    delta_yaw: float | None = None,
    gripper: float = 1.0,
    max_steps: int = 40,
    tol: float = 0.02,
    step_clip: float = 0.1,
) -> Rotation
    Rotate the gripper about the world z axis, keeping its position.

    Give exactly one of ``target_yaw`` (absolute, rad) or ``delta_yaw``
    (relative, rad).  ``state().eef_yaw`` reports the current yaw.
    ``gripper`` defaults to +1 (keep holding).

robo.rotate_pitch(
    *,
    target_pitch: float | None = None,
    delta_pitch: float | None = None,
    gripper: float = 1.0,
    max_steps: int = 40,
    tol: float = 0.02,
    step_clip: float = 0.1,
) -> Rotation
    Tilt the gripper about the world x axis, keeping position and yaw.

    Pitch 0 points the gripper straight down; +pi/2 points it along world
    +y, -pi/2 along -y.  Use before threading into an opening whose front
    faces along y.  Give exactly one of ``target_pitch`` or ``delta_pitch``.

robo.set_gripper(gripper: float, steps: int = 5) -> Gripper
    Hold the current pose and drive the gripper for ``steps`` simulator steps.

    ``set_gripper(+1, steps=8)`` right after a successful pick firms the
    grip before carrying (use ``steps<=5`` for a laterally weak can).

robo.release(max_steps: int = 20) -> Release
    Open the gripper in place for up to ``max_steps`` steps and let go.

    Returns as soon as the task predicate fires (``terminated``), so a
    successful placement usually ends the episode here.

robo.pi0_pick(
    prompt: str,
    *,
    max_chunks: int = 24,
    lift_thresh: float = 0.05,
    gripper_closed_thresh: float = 0.06,
    gripper_open_thresh: float = 0.0,
    descent_thresh: float = 0.1,
) -> Pick
    Let the frozen Pi0.5 VLA perform a closed-loop grasp.

    The VLA runs ``prompt`` for up to ``max_chunks`` action chunks (each a
    short burst of steps) and stops early once the gripper has descended by
    ``descent_thresh``, closed on something (opening in
    ``[gripper_open_thresh, gripper_closed_thresh)``) and risen by
    ``lift_thresh``.  Use it for the grasp: pre-position the gripper above
    the target, keep the prompt short (``"pick up the red mug"``), use a
    modest chunk budget (20 is the usual choice) and verify the grasp from
    the lift, the gripper closure and the images; then script the carry
    and release yourself.

    Returns:
        Pick; ``success`` is a heuristic - verify with ``state()`` and the
        wrist image before carrying.

robo.pi0_doubled(prompt: str, *, max_chunks: int = 20) -> Contact
    Let the frozen Pi0.5 VLA perform a contact skill that is not a pick.

    For knobs, buttons, drawers, doors and short pushes (``"turn on the
    stove"``, ``"open the top drawer"``).  Runs up to ``max_chunks`` chunks
    or until the task predicate fires.  ``success`` only mirrors that
    predicate; check the state or an image to judge an intermediate step.

Result types (frozen dataclasses; access fields as attributes):

class State(
    step: int,
    task: str,
    eef_pos: tuple[float, float, float],
    eef_quat: tuple[float, float, float, float],
    gripper_qpos: tuple[float, float],
    gripper_opening: float,
    object_names: tuple[str, ...],
    terminated: bool,
    truncated: bool,
)
    Proprioceptive snapshot of the robot at one recorded step.

    step: index of the recorded step (0 = initial scene).
    task: the task instruction given by the benchmark (authoritative).
    eef_pos: end-effector (gripper) position [x, y, z] in the world frame, metres.
    eef_quat: end-effector orientation quaternion [x, y, z, w].
    gripper_qpos: the two finger joint positions.
    gripper_opening: finger separation proxy, |q0| + |q1|.  About 0.08 when
        fully open, about 0.0 when closed on nothing; roughly 0.01-0.05 means
        the fingers closed on an object.
    object_names: names of the movable objects in the scene (no poses).
    terminated: the benchmark's success predicate has fired.
    truncated: the simulator step budget is exhausted.
    .eef_yaw -> float: World-frame yaw of the gripper in radians (as used by rotate_wrist).
    .eef_pitch -> float: Gripper tilt in radians (0 = pointing straight down; see rotate_pitch).

class Move(
    target_xyz: tuple[float, float, float],
    final_eef_pos: tuple[float, float, float],
    final_dist_m: float,
    reached: bool,
    steps_used: int,
    max_steps: int,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``move_to``.

    target_xyz: the commanded target.
    final_eef_pos: where the gripper actually stopped.
    final_dist_m: distance between the two, metres.
    reached: ``final_dist_m <= tol``.  False means the servo stalled (IK limit,
        collision, or step budget) - re-plan instead of assuming you arrived.
    steps_used / max_steps: simulator steps consumed / allowed.
    terminated / truncated: episode flags after the move.

class MovePose(
    final_eef_pos: tuple[float, float, float],
    final_dist_m: float,
    final_pitch: float,
    reached: bool,
    steps_used: int,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``move_pose`` (position and orientation servo).

    reached: position within ``tol`` (orientation errors are not folded in).
    final_pitch: gripper pitch after the move, radians.

class Rotation(
    start: float,
    target: float,
    final: float,
    final_err: float,
    reached: bool,
    steps_used: int,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``rotate_wrist`` / ``rotate_pitch``; angles in radians.

    reached: ``abs(final_err) <= tol``.

class Gripper(gripper: float, steps: int, terminated: bool, truncated: bool)
    Outcome of ``set_gripper``: the command that was held and for how long.

class Release(
    steps_used: int,
    start_gripper_opening: float,
    peak_gripper_opening: float,
    final_gripper_opening: float,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``release``.

    peak_gripper_opening: widest finger separation seen while opening.
    terminated: True if letting go satisfied the task predicate.

class Pick(
    success: bool,
    chunks_used: int,
    max_chunks: int,
    peak_lift_m: float,
    min_gripper_opening: float,
    final_gripper_opening: float,
    terminated: bool,
    truncated: bool,
    diagnostics: dict[str, Any],
)
    Outcome of ``pi0_pick``.

    success: the VLA's lift heuristic fired (descended, then rose by
        ``lift_thresh`` with the fingers closed on something).  It is a hint,
        not proof: confirm with ``state().gripper_opening`` and the wrist image.
    chunks_used / max_chunks: VLA action chunks consumed / allowed.
    peak_lift_m: how far the gripper rose after its lowest point.
    min_gripper_opening / final_gripper_opening: finger separation statistics.
    terminated: the task predicate fired during the pick.
    diagnostics: raw threshold bookkeeping from the primitive.

class Contact(
    success: bool,
    chunks_used: int,
    max_chunks: int,
    terminated: bool,
    truncated: bool,
)
    Outcome of ``pi0_doubled``.

    success: mirrors ``terminated`` only.  For an intermediate contact skill
        (knob, drawer, door) ``False`` does not mean the contact failed;
        inspect the state or an image.

class Segment(
    found: bool,
    world_xyz: tuple[float, float, float] | None,
    score: float | None,
    box: Any,
    camera: str,
    step: int,
    overlay_artifact: str | None,
    error: str | None,
)
    Outcome of ``segment``.

    found: a mask above ``min_score`` was produced.
    world_xyz: median world coordinate of the mask (surface point, metres), or
        None when not found / no valid depth.
    score: SAM3 confidence in [0, 1] (brand nouns score ~0.05, colour+shape
        phrases ~0.7).
    box: mask bounding box as reported by SAM3, or None.
    overlay_artifact: name of the saved overlay image (``artifact()`` loads it).
    error: why nothing was found, or None.

class BackProjection(
    world_xyz: tuple[float, float, float],
    pixel: tuple[int, int],
    camera: str,
    resolution: str,
    step: int,
    image_size: tuple[int, int],
)
    Outcome of ``back_project``: the world point under one pixel.

class RegionCenter(
    center_xyz: tuple[float, float, float],
    median_xyz: tuple[float, float, float],
    n_valid: int,
    camera: str,
    step: int,
)
    Outcome of ``region_center``.

    center_xyz: midpoint of the region's world x/y extent with median z.
    median_xyz: per-axis median of the same pixels.
    n_valid: pixels that survived the depth and z-band filters.

Exceptions:
    ToolError(RuntimeError): a call was rejected or a service failed; catch it if you have a fallback.
    ValueError: an argument was invalid (e.g. a planar move over 0.30 m) - nothing was executed.
    EpisodeFinished(BaseException): the episode is over (reason: terminated | truncated | timeout | cancelled); return.