Drag the skeleton. Scrub the motion.
Orange spheres = what the headset actually measures (head + two hands). Dark joints = what the IK solver infers. Toggle the solver off to see how little raw data there really is.
Why a headset is secretly a mocap studio
What is measured
A standalone headset tracks its own 6-DoF pose (position + orientation) via inside-out SLAM cameras at 60–90 Hz, and hand poses via controllers or computer-vision hand tracking. That is 3 rigid poses — 18 numbers per frame.
What is inferred
Elbows, shoulders, spine, hips, knees, ankles. An IK solver (or a learned neural pose prior trained on real mocap) picks the most plausible skeleton consistent with those 3 anchors plus constraints like joint limits and floor contact.
Why robots care
Imitation learning needs demonstrations of the whole body moving with intent: how the torso leans before a reach, how weight shifts in a squat. Hand-only data misses the postural strategy humanoid robots must copy.
The economics
An optical mocap studio runs $50k–$250k with per-session labor. A consumer headset costs ~$500 and captures anywhere. Slightly noisier data, but thousands of times more of it — and scale usually wins in robot learning.
Worked example: solving one elbow
Two-link IK is just the law of cosines. Say the upper arm is 0.30 m, the forearm 0.28 m, and the tracked wrist sits 0.50 m from the shoulder. The interior elbow angle φ must satisfy:
cos φ = (0.30² + 0.28² − 0.50²) / (2 · 0.30 · 0.28)
cos φ = (0.090 + 0.0784 − 0.250) / 0.168 = −0.486
φ = arccos(−0.486) ≈ 119°
That fixes the bend, but the elbow can still swivel around the shoulder–wrist axis — a full circle of equally valid solutions. Real systems break the tie with a "swivel prior": elbows point down-and-out because that is what human elbows overwhelmingly do. Neural IK models learn that prior from mocap datasets like AMASS instead of hand-coding it.
The pipeline, end to end
1. Capture: headset streams head + hand poses while a person does a real task (fold laundry, load a dishwasher). 2. Solve: full-body IK reconstructs a 17–24 joint skeleton per frame. 3. Retarget: the human skeleton is mapped onto the robot's kinematics (different limb lengths, different joint limits). 4. Train: behavior-cloning or diffusion-policy models learn to reproduce the trajectories. Each stage loses a little fidelity — which is why clean, intentful source motion matters more than any single algorithm.