We present TeleOpBench, a simulation-based benchmark for bimanual dexterous teleoperation, and evaluate four representative teleoperation modalities across multiple robot platforms (row 1). Real-robot experiments (row 2) demonstrate four teleoperation capabilities. Our teleoperation pipelines support fine-precision manipulation in the real world—for example, the left hand grasps a block while the right hand simultaneously inserts a smaller block (row 3)—and can execute long-horizon sequences, such as retrieving a tomato-laden plate from a microwave with the right hand and transferring the tomatoes to a table with the left (rows 4 and 5).

This paper makes the following contributions:

1. We introduce a dedicated benchmark, TeleOpBench, for dual-arm dexterous teleoperation, enabling rigorous, fair, and comprehensive comparisons across competing systems.
2. We implement four representative teleoperation pipelines—motion-capture, VR controllers, upper-body exoskeletons, and vision-only within a single modular framework.
3. Extensive experiments on both TeleOpBench and a real dual-arm platform reveal a strong correlation between simulated and physical performance, substantiating the benchmark's fidelity and practical value.

We employ NVIDIA Isaac Sim as our simulation platform because its high-performance PhysX engine and photorealistic renderer enable the construction of environments that closely approximate real-world conditions. Each scene features a humanoid robot fitted with bimanual dexterous hands and the task-relevant objects; operators are instructed to execute the required manipulations exactly as specified. For every trial, we record both task success and completion time, which together constitute our primary performance metrics.

30 High-Fidelity Task Environments

We implement four representative teleoperation pipelines—monocular vision, MoCap, VR, and exoskeleton— under a unified, modular interface.

1.Vision-based

The system first calibrates once in a neutral T‑pose, using SMPL to derive body shape β and link-scale factors s that align human and robot kinematics. During operation, SMPLer-X streams the operator's upper-body pose, which is rescaled by s and solved with PINK IK for arm-and-wrist motion, while MediaPipe key-points refined by Dex-Retargeting drive precise finger control. Decoupling limb and hand estimation yields robust, real-time teleoperation from pure vision input.

2.MoCap-based

The inertial-MoCap pipeline uses an Xsens MVN suit (23 IMUs) plus Manus Metagloves. After a one-time calibration, the MVN stream provides the 6-DoF pose of 23 body segments, while each glove outputs 20 finger-joint DoFs. Raw limb data are first transformed from the MVN's global frame to the robot frame (pelvis origin, forward +X, vertical +Z).
A joint-specific, real-time rescaling module then compensates for human-robot link-length mismatches before Closed-Loop Inverse Kinematics (CLIK) solves the robot's arm-and-wrist poses. For the hands, the glove's MCP, PIP, DIP and ab/adduction angles are mapped directly—subject to the dexterous hand's joint limits—yielding accurate, low-latency replication of both limb and finger motions.

3.VR-based

The VR-based teleoperation system includes two main components:

1. Upper-body Limb Motion Control

For upper-body limb motion control, the Apple VisionPro is utilized for hand, wrist, and head tracking, adhering to the OpenXR coordinate system. Wrist and head poses are initially transformed into the robot's coordinate frame. The wrist offset relative to the head is then converted into an offset relative to the pelvis. Only the wrist translation data is fed to an IK algorithm based on Pink, which computes all degrees of freedom except for finger joints.

2. Hand Control

For hand control, to enhance manual dexterity across different teleoperators, the distal phalanx lengths of each operator's fingers are measured and scaled proportionally to match the corresponding robotic finger segments. Subsequently, vector-based optimizers are employed, following the OpenTelevision approach, to generate robot-hand joint commands within the dexterous-retargeting framework of AnyTeleop.

4.Exoskeleton-based

This exoskeleton-based teleoperation framework creates isomorphic systems customized to replicate a target humanoid's upper body kinematics, based on HOMIE principles. Servo-driven joints ensure real-time synchronization of operator and robot movements. Integrated motion-sensing gloves with Hall-effect sensors provide 15 DoF per-hand tracking. By directly mapping operator kinematics to the humanoid's joints, this method bypasses inverse kinematics (IK) approximations, thereby eliminating algorithmic errors and enhancing operational bandwidth and positional accuracy.

1. Isomorphic Exoskeleton for Unitree G1.

2. Isomorphic Exoskeleton for Fourier GR-1.

3. Isomorphic Exoskeleton for Unitree H1-2.

From top to bottom, we illustrate the four teleoperation modalities executing the following tasks: ball trashcan, pen brushpot, ball bimanual, and pot bimanual.

1. Simulation Results

We select ten representative tasks of varying difficulty from the TeleOpBench: (1) push_cube, (2) pich_cube, (3) pick_place_cube, (4) uprear_cup, (5) ball_trashcan, (6) ball_mug, (7) ball_bimanual, (8) pot_bimanual, (9) pot_tomato_plate, and (10) pen_brushpot. Full task descriptions refer to Table ref{tb:task}. A user study involving four participants was conducted; Task-level success rates and completion times are summarized quantitatively in Table

2. Real-world Results

We reproduce the task suite on physical robots and evaluate all four teleoperation pipelines with the identical metric suite; the resulting quantitative scores are summarized in Table.

Figure below presents completion-time curves for simulation and real world. Note that Tasks with one teleoperation success rate below 20% are excluded from the plotted curves to ensure the reliability of the curves. The two domains exhibit a strong positive correlation: the vision-tracking interface consistently requires the longest execution time (blue curve), the inertial- MoCap pipeline is the fastest (red curve), and the VR and exoskeleton interfaces cluster in between. This close alignment between simulated and real-world performance confirms that TeleOpBench reliably predicts practical outcomes and therefore offers substantial utility to the community.

Sim

Real