RoboTok

A Scalable Data Engine for Internet Demonstration Video
Retrieval and Dexterous Manipulation Learning

Anonymous Authors

Code Data and Models
RoboTok is an internet-scale data engine for robot manipulation learning. It organizes internet human demonstration videos into a torso-relative 3D hand trajectory embedding space and retrieves clips with similar hand motions for a given query demonstration.

Abstract

Robot learning increasingly depends on broad and diverse demonstrations, yet collecting robot data remains expensive and poorly suited to covering the wide range of real-world tasks. To address this bottleneck, we introduce RoboTok, a scalable data engine that, given a query human manipulation video, retrieves manipulation-relevant human demonstrations from internet videos for training dexterous robot policies. Specifically, we learn a latent motion space from 3D hand trajectories expressed in estimated actor-centered reference frames. This representation enables manipulation behaviors to be compared across variations in camera viewpoint, scene appearance, and actor occlusions, while remaining compact enough for efficient search and continual indexing over internet video collections. We evaluate RoboTok against existing robot-data retrieval approaches on retrieval metrics and downstream robot policy performance. Our results show that RoboTok retrieves more relevant manipulation demonstrations and improves downstream task success, establishing hand-pose trajectory-aware retrieval as a way to make web video a scalable and continuously growing source of supervision for robot learning.

Method

RoboTok converts internet human demonstration videos into torso-relative 3D hand trajectories and learns a compact embedding supervised by DTW-based motion similarity. This representation enables manipulation behaviors to be compared across variations in camera viewpoint, scene appearance, and actor occlusions. At query time, cosine-similarity search retrieves related manipulation demonstrations without computing DTW over the full corpus.

Internet videos → Clip filtering → Data extraction → Hand motion-based retrieval → Robot policy training

AData processing pipeline
Data processing pipeline: clip filtering, 3D hand reconstruction, then torso estimation and actor-centered representation
BDTW-aligned trajectory encoder
DTW-aligned trajectory encoder: the corpus encoded into a latent motion space, and a query retrieved from it

Retrieval Demo (Interactive)

Unlike appearance-based retrieval methods, RoboTok retrieves video clips with similar hand-pose trajectories across different activities, scenes, and camera viewpoints. We use a trained model to estimate a human torso-centered reference frame using only the 3D hand trajectory, which is effective even when the actor is not directly visible.

10 example query videos and their corresponding retrieved videos per method are shown below. An interactive viewer of extracted egocentric 3D hand trajectories is also provided.

Retrieval Results

Using canonicalized 3D hand trajectories, RoboTok retrieves manipulation demonstrations on both the in-domain RoboTok evaluation corpus and the out-of-domain AssemblyHands [1] corpus, outperforming all evaluated retrieval baselines across every reported metric.

In-domain retrieval on the RoboTok evaluation corpus

Method mAP@10 ↑
[6]		 nDCG@10 ↑
[7]		 CKNNA@10 ↑
[8]		 MRR@10 ↑
[9]		 Kendall τ ↑
[10]		 DTW cost@10 ↓
[11]
Random0.0000.0000.0000.0000.0004.778
Flow (CoRL 2024) [3]0.0010.0020.0010.0030.0036.795
HAND (ICRA 2026) [4]0.0010.0040.0020.0090.0124.508
STRAP (ICLR 2025) [5]0.0100.0280.0270.0670.0143.901
RoboTok0.3360.5340.4530.7890.4871.245
GT (upper bound)1.0001.0001.0001.0001.0001.069
Table 1. In-domain retrieval on the RoboTok evaluation corpus. The corpus contains 100,000 internet human clips. We train the retrieval model on 90,000 clips and use the remaining 10,000 clips as unseen evaluation queries. The evaluation database contains all 100,000 clips, with the query itself excluded from retrieval. For each query, we define its 10 nearest clips under the DTW metric as the ground-truth relevant set. We evaluate each method by comparing its ranked retrieval results with the relevant set and ranking defined by DTW.

Out-of-domain retrieval on AssemblyHands [1]

Method mAP@5 ↑
[6]		 nDCG@5 ↑
[7]		 CKNNA@5 ↑
[8]		 MRR@5 ↑
[9]		 Kendall τ ↑
[10]		 DTW cost@5 (m) ↓
[11]
Random0.0030.004−0.0010.0130.0061.911
Flow (CoRL 2024) [3]0.0130.0260.0180.0590.0901.850
HAND (ICRA 2026) [4]0.0210.0430.0260.0960.0801.830
STRAP (ICLR 2025) [5]0.1330.2290.1810.3970.1571.304
RoboTok0.2610.3870.3360.6210.2681.095
GT (upper bound)1.0001.0001.0001.0001.0000.966
Table 2. Out-of-domain retrieval on the AssemblyHands [1] corpus. The evaluation corpus contains 831 two-hand assembly clips with sensor-grade 3D hand annotations. During evaluation, each clip serves once as a query, and the remaining 830 clips form its candidate set. For each query, we define its five nearest clips under the DTW metric as the ground-truth relevant set. We evaluate each method by comparing its ranked retrieval results with the relevant set and ranking defined by DTW.

Simulation Tasks

RoboTok-retrieved demonstrations help robot policies learn to complete simulation tasks more successfully than demonstrations retrieved by baseline methods. The evaluation rollout videos below compare the resulting policy behaviors. Only proprioception and fingertip force are used as inputs.

Two cameras are provided to show rollouts from different angles, but are not used as inputs to policies.

Simulation Task Results

RoboTok-guided policies achieve the highest success rates on all three modified VTDexManip [2] tasks, on both seen and unseen objects. Tasks are modified to be larger exploration problems by unrestricting hand motion in 3D space and removing many dense task rewards.

Modified VTDexManip [2] tasks

Tasks Split Base Random Flow (CoRL 2024) [3] HAND (ICRA 2026) [4] STRAP (ICLR 2025) [5] RoboTok
BottleCap TurningSeen0.1±0.10.2±0.11.9±1.859.5±9.059.0±7.377.3±3.5
Unseen0.0±0.00.1±0.10.0±0.027.0±6.025.0±5.938.1±5.5
Faucet ScrewingSeen1.0±0.65.3±4.80.0±0.06.8±4.80.0±0.044.8±4.1
Unseen1.7±1.71.0±1.00.3±0.32.6±1.70.0±0.010.9±2.8
Lever SlidingSeen0.8±0.317.2±6.814.4±7.319.5±4.88.4±4.179.3±5.3
Unseen2.1±2.118.8±11.919.6±12.225.2±11.86.5±3.458.3±7.5

Real World Tasks

Real-robot rollouts for the three tasks, each with three object variants. Policies guided by RoboTok are compared with policies guided by HAND [4] and STRAP [5], the two best baselines from simulation. Every clip is captured from a fixed camera; bottle-cap and faucet clips are full episodes, and lever clips run from the grasp until the hand lifts away.

Smooth black tape is added to help see object motions and does not help with task completion.

Real World Task Results

Real-world success rate (%) for policies guided by HAND [4], STRAP [5], and RoboTok.
Bars show the mean over three objects, with markers showing per-object success over 10 rollouts.

References

  1. Takehiko Ohkawa, Kun He, Fadime Sener, Tomáš Hodaň, Luan Tran, and Cem Keskin. AssemblyHands: Towards egocentric activity understanding via 3D hand pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12999–13008, 2023. assemblyhands.github.io
  2. Qingtao Liu, Yu Cui, Zhengnan Sun, Gaofeng Li, Jiming Chen, and Qi Ye. VTDexManip: A dataset and benchmark for visual-tactile pretraining and dexterous manipulation with reinforcement learning. In International Conference on Learning Representations (ICLR), 2025. lqts.github.io/VTDexManip
  3. Li-Heng Lin, Yuchen Cui, Amber Xie, Tianyu Hua, and Dorsa Sadigh. FlowRetrieval: Flow-guided data retrieval for few-shot imitation learning. In Conference on Robot Learning (CoRL), 2024. flow-retrieval.github.io
  4. Matthew Hong, Anthony Liang, Kevin Kim, Harshitha Rajaprakash, Jesse Thomason, Erdem Bıyık, and Jesse Zhang. HAND me the data: Fast robot adaptation via hand path retrieval. In IEEE International Conference on Robotics and Automation (ICRA), 2026. liralab.usc.edu/handretrieval
  5. Marius Memmel, Jacob Berg, Bingqing Chen, Abhishek Gupta, and Jonathan Francis. STRAP: Robot sub-trajectory retrieval for augmented policy learning. In International Conference on Learning Representations (ICLR), 2025. weirdlabuw.github.io/strap
  6. Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference on Computer Vision (ECCV), 2020. arxiv.org/abs/2003.08505
  7. Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446, 2002. doi.org/10.1145/582415.582418
  8. Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. In International Conference on Machine Learning (ICML), 2024. arxiv.org/abs/2405.07987
  9. Ellen M. Voorhees. The TREC-8 question answering track report. In Proceedings of the 8th Text REtrieval Conference (TREC-8), pp. 77–82, 1999. trec.nist.gov/pubs/trec8/papers/qa8.pdf
  10. Maurice G. Kendall. A new measure of rank correlation. Biometrika, 30(1–2):81–93, 1938. doi.org/10.1093/biomet/30.1-2.81
  11. Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26(1):43–49, 1978. doi.org/10.1109/TASSP.1978.1163055

BibTeX

@article{anonymous2026robotok,
  title     = {RoboTok: A Scalable Data Engine for Internet Demonstration
               Video Retrieval and Dexterous Manipulation Learning},
  author    = {Anonymous},
  note      = {Under review},
  year      = {2026}
}