How Browser-Based Hand Tracking Actually Works

When you hold a hand up to a Motion Arcade cabinet, a machine learning model running inside your browser locates it and returns 21 landmarks: the wrist, three joints per finger, and each fingertip, every one as an x and y position in the camera frame. We use MediaPipe, Google's open on-device vision library, which ships the model to your browser like any other page asset and runs it locally with WebAssembly and GPU acceleration. There is no plugin and nothing to install; the model is just part of the page.

Head-controlled cabinets use the face sibling of that model. It returns a dense mesh of facial landmarks plus expression scores, which is how a game can read something as specific as "the eyebrows just went up" or "the mouth is now open" without ever interpreting the image itself beyond those numbers.

The naive architecture for camera games is to stream frames to a server, run a big model there, and send the results back. That version is unshippable for an arcade: a round trip adds tens to hundreds of milliseconds of lag, costs someone money per frame, and means your camera feed is on someone else's computer. Running the model locally solves all three at once. The latency between your hand moving and the mallet moving in Air Hockey is bounded by your own hardware, and the video never leaves it (more on that in the privacy guide).

Here is the counterintuitive part: the games do not run detection on your full camera resolution. Your webcam might deliver 1280 by 720 pixels, but the tracking pipeline downscales each frame to about 180 pixels tall before the model sees it, and it caps detection at roughly 30 passes per second regardless of how fast the camera delivers frames. Landmark models do not need many pixels to find a hand; they need consistent ones. Downscaling makes detection cost nearly identical on a gaming laptop and a mid-range phone, which is why the same cabinets run on both.

Meanwhile the game itself renders at your display's full frame rate. Tracking at 30 updates a second under a game drawing at 60 would feel like a slideshow if the raw positions were used directly, which leads to the real craft of the thing.

Raw landmarks jitter. Fingers tremble, detection wobbles by a pixel or two each pass, and naive smoothing fixes the jitter by making everything feel like it is dragged through syrup. The cabinets use a few different tools depending on what the input drives. Paddle games apply an exponential smoothing that eases the paddle toward the hand's reported position each frame, tuned so slow drifts are steadied but a deliberate flick still lands. Air Hockey layers a One Euro filter on top, a small algorithm whose whole idea is to smooth aggressively when your hand moves slowly (when jitter is visible) and barely at all when it moves fast (when lag would be felt). Fruit Slice even extrapolates briefly between detections, projecting where your fingertip is headed for up to 60 milliseconds so the blade trail never stutters mid-swing, and it tests slices as a swept line segment between the last two positions so a fast swipe cannot tunnel through a fruit between frames.

Different games consume different slices of the same data. Fruit Slice tracks the index, middle, and ring fingertips of both hands as blades. Motion Pop treats all ten fingertips as poppers. Air Hockey and Hand Pong collapse the whole hand to its center point, because a mallet wants stability, not finger detail. The Zen Scale measures the angle of the line between your eyes to read head roll. Brick Breaker in head mode just follows the center of your face box left and right. Inferno Drop ignores hands entirely and reads expression scores: mouth open fires, eyebrows throw, a smile triggers Overdrive. Same camera, same models, eight different controllers.

The fastest way to feel all of this is Motion Pop, which is deliberately nothing but the tracking loop: fingertips in, pops out. If the tracking feels off in any cabinet, the fix is almost always physical, not digital; see setting up a play space that tracks well.