TL;DR What novices see shapes the demonstrations they collect, so the operator's view and the policy's observation are worth designing together.

Overview

Imitation learning increasingly relies on novice operators, whose demonstrations fall short of experts' in number and inter-trial consistency. Operators act on what they are shown, yet how the rendering of the robot affects their demonstrations, and the policies trained on them, has not been examined. We study this question with PointHand, a teleoperation interface that renders a multi-fingered robot hand as sparse keypoints at its fingertips and joints, the same keypoints a policy can observe. In a simulated mug-in-box task, 18 novices and 4 experts collected 2,200 trials under three renderings: the real shape of the hand, all keypoints emphasized, or the thumb and index finger alone emphasized. The novices shown the thumb and index finger emphasized collected 441 successful demonstrations from 600 trials, against 320 for those shown the real shape. We then trained diffusion policies on these demonstrations, replacing the scene point cloud and proprioceptive state they observe with the hand keypoints and two object keypoints. Within 400 steps these demonstrations went from the weakest of the three novice conditions under the point-cloud policy (46.3%) to the strongest under the keypoint policy (57.3%), a gain of +11.0 percentage points in all five seeds. Those collected with all keypoints emphasized changed by at most +0.7 percentage points, and in no more than three seeds. The value of a demonstration therefore depends on the rendering shown during collection and on what the policy observes together. A shared representation can pay off on both sides, in the number of successful demonstrations and in the success rate of the policy trained on them.

PointHand: The Hand as Sparse Keypoints

PointHand replaces the robot hand shown to the operator with sparse keypoints at the fingertips and at the joints that bend each finger, the same keypoints a policy can observe. Emphasized keypoints have a per-finger color that brightens when the finger comes close to the object, so the rendering also carries a contact cue. We compare two keypoint renderings with the real shape of the hand. The colors below are the ones used for each condition in the result graphs.

The robot hand rendered as its real shape
Baseline
real shape
The robot hand rendered as keypoints with all keypoints emphasized
KP-Standard
all keypoints emphasized
The robot hand rendered as keypoints with only the thumb and index finger emphasized
KP-Salient
thumb and index finger emphasized
Fingertip keypoints are spheres and intermediate joint keypoints cubes. Keypoints that are not emphasized, and the wrist keypoint, which the policy does not receive, are gray. The experts used the real-shape rendering (Expert, brown in the graphs).

Experiment

We investigated how the demonstrations, and the policies trained on them, differ between PointHand and the real shape of the hand in a simulated mug-in-box task.

Study 1: collecting demonstrations. 18 novices, six per rendering, and 4 experts shown the real shape each teleoperated the robot hand for 100 trials, 2,200 trials in total.

Study 2: training policies. From the successful demonstrations of each condition we trained two diffusion policies on the same demonstrations, which record the keypoint positions whether or not they were shown to the operator. DP3 observes the scene point cloud and the proprioceptive state. DP-KP observes the hand keypoints and two object keypoints instead. Both share the diffusion head and the training procedure.

Observation encoders of DP3 and DP-KP Observation encoding of the two policies. Top: DP3 uses the encoder of 3D Diffusion Policy (Ze et al., RSS 2024) on the scene point cloud and the state. Bottom: DP-KP uses the encoder of Point Policy (Haldar and Pinto, CoRL 2025) on the hand and object keypoints.

Results

Study 1: demonstrations collected by the operators

Novices (six per condition) shown KP-Salient succeeded in 73.5% of their trials on average, against 53.3% for those shown the real shape, and collected 441 successful demonstrations from 600 trials, against 320.

Success rate during collection per condition

Study 2: success rate of the policies trained on them

The same KP-Salient demonstrations were the weakest of the three novice conditions under DP3 within 400 steps and the strongest under DP-KP.

Collection condition DP3 DP-KP
200400600 200400600
Baseline24.747.753.325.756.365.0
KP-Standard22.754.061.719.053.762.3
KP-Salient24.746.352.339.757.365.0
Expert (reference)39.062.065.744.370.077.0
Simulated success rate [%] within 200, 400, and 600 steps, mean over five seeds. Bold marks the highest value among the three novice conditions in each column. Simulated success rate of DP3 and DP-KP per collection condition

Discussion

Choose the rendering and the policy's observation together. The same KP-Salient demonstrations were the weakest of the three novice conditions under DP3 within 400 steps (46.3%) and the strongest under DP-KP (57.3%).

A shared representation pays off on both sides only when it is designed well. KP-Salient gave the lowest Mental Demand and 441 successful demonstrations of 600 trials (320 with the real shape), and trained the strongest DP-KP within 200 and 400 steps. KP-Standard shared the same keypoints with the policy, yet its operators carried the highest Mental Demand and its demonstrations were the weakest under DP-KP.

Contact

contact@sinicx.com

Citation

@article{yamaguchi2026novice,
  title={What Novice Operators See Changes What the Policy Learns: A User Study of Keypoint Hand Visual Representation for Teleoperated Demonstration},
  author={Yamaguchi, Amane and Kamiguchi, Shohei and Hamaya, Masashi and Yoshida, Shigeo and Narumi, Takuji and Nishimura, Mai},
  journal={arXiv preprint arXiv:XXXX.XXXXX},
  year={2026}
}