Two people captured at once, each driving their own avatar — the wall is the running Unreal editor, not a playback
第1章Chapter 01 — The problem
Capture that fits in a case
Marker-based capture works, but it costs a studio: a dedicated volume, an optical array, suits, calibration time, a technician. It does not travel, and it does not scale to anyone who is not already set up for it. We wanted capture that could be carried into a room, plugged in, and used by someone in ordinary clothes.
The binding constraint was where inference had to run. Streaming frames to a workstation or the cloud adds latency, and in XR latency between your body and your avatar breaks embodiment immediately. It had to happen on the device, in the room, in real time.
第2章Chapter 02 — What I built
Four views, one skeleton
Four synchronised RGB cameras feed pose inference running on a Jetson Orin NX. Each view produces 2D keypoints; triangulation lifts them to 3D; a Kalman filter stabilises the result into a 17-joint skeleton, which streams into Unity and Unreal through a custom plugin.
第3章Chapter 03 — The hard part
Getting it off the workstation
The pipeline was developed against an RTX 3080. Moving it onto a Jetson Orin cost almost everything: four cameras with every model running through ONNX landed at 3–4 FPS, which is not a slow system — it is an unusable one. Nothing about the approach was wrong; it simply had no route to real time on the hardware it had to ship on.
The first pass was precision and runtime. Converting the models to FP16 and baking them to TensorRT for the Orin specifically took it to around 20 FPS — a large win, and still short. The second was batching: rather than four cameras taking turns through the network, all views and all inference passes were batched into single calls, so the GPU stopped idling between frames it could have been working on.
The last one was the least obvious and mattered most. The detection and pose models had been built with dynamic input shapes, which meant TensorRT could not commit to an optimised plan and re-planned as shapes moved. Rebaking every engine to fixed static input and output shapes removed that entirely, and the pipeline settled at 40–55 FPS on a module you can carry in a bag.
第4章Chapter 04 — The result
Two people, four webcams, no workstation
Two people captured simultaneously from four ordinary webcams, on a module carried to site, driving avatars live in both Unity and Unreal through a plugin the rest of the team could use without touching the CV stack. A One Euro filter smooths the skeleton at the engine end, so what an animator sees is steadier than the raw frame rate suggests.
Accuracy sits at roughly 87% across a fixed list of test poses run in real time. The failures are honest and specific: limbs that leave the camera volume entirely, and poses that fold the body over itself — curling up, hiding a limb behind the torso — where four views stop being four views.