Build Hand Controlled Visuals in the Browser with Claude Code

Hand controlled visuals look like the kind of thing that needs a studio, a depth camera and a week of work. In reality it is a normal webcam, a free model from Google that runs inside the browser, and a canvas. The tracking is solved. What makes these pieces feel good is how you turn a moving hand into a few clean numbers, and that is the part most people skip.

I build interactive pieces like this with Claude Code, and the method below is the one I trust: get the signals right first, then attach any visual you want to them.

How it actually works

Google's MediaPipe Hand Landmarker takes a video frame and returns 21 landmarks per hand, each with x, y and z coordinates, plus whether it is a left or right hand. It runs locally in the browser through WebAssembly, so the camera frames never have to leave the laptop. The model file and the library are downloaded from a CDN when the page loads, which is worth telling your viewers.

Those 21 points are too noisy to drive a visual directly. You reduce them to four or five signals:

• Position: the palm center, an average of the wrist (point 0) and the four knuckles (points 5, 9, 13, 17).
• Openness: the average distance from the four fingertips (8, 12, 16, 20) to the wrist, divided by palm size (wrist to point 9). Dividing by palm size means it works whether your hand is close to the camera or far away.
• Pinch: the distance between the thumb tip (4) and the index tip (8), also divided by palm size.
• Rotation: the angle of the line from the wrist to point 9.

Every effect you have seen is some combination of those. A web of nodes that blooms is openness mapped to scale. A video that crumples is openness mapped to playback time. A 3D object you can spin is rotation mapped to a transform.

What you need

• Claude Code, set up with the official quickstart.
• Chrome or Safari on a laptop with a webcam. In app browsers often block the camera.
• An empty project folder.

One rule catches almost everyone. Browsers only give camera access in a secure context, which means https or localhost. Double clicking an HTML file is unreliable, so always serve it locally.

Step 1: build the signal layer first

Start with a page that shows your webcam, draws the skeleton and prints the numbers. No visual yet. Paste this into Claude Code inside the empty folder:

Build a single index.html that tracks one hand from the webcam and shows the
signals I will use later. No visuals beyond a debug view.

Tracking: use MediaPipe Tasks Vision (@mediapipe/tasks-vision) Hand Landmarker
from jsDelivr. Pin one exact version for both the JS bundle and the wasm folder.
Check the current official web guide before writing code. VIDEO running mode,
numHands 1, GPU delegate with a CPU fallback. Only call detectForVideo when
video.currentTime has changed.

Camera: a Start button that explains what the camera is used for, then calls
getUserMedia. A Stop button that stops every media track. Mirror the video.

Signals, each smoothed with exponential smoothing and shown live on screen:
position (palm center), openness 0 to 1, pinch 0 to 1, rotation in degrees,
and hand present true or false. Normalize distances by palm size.

States: loading model, camera denied, no hand, tracking. Add a mouse fallback
that fakes the signals (x, y for position, scroll wheel for openness).

Serve it on 127.0.0.1 with a free port and give me the URL. Open it with any
browser tool you have, read the console, and fix errors before you hand it back.
Do not claim you tested the webcam. List the domains the page downloads from.

Now play with it. Open and close your hand slowly and watch openness. If it jumps between 0.3 and 0.7 while your hand is still, you need more smoothing or better light. Fix this here, because every visual you add later inherits the jitter.

Step 2: calibrate, do not guess

Hands are different sizes and people hold them at different angles. Hard coded thresholds feel broken for half your audience. Ask Claude to add a two second calibration:

Add calibration: ask me to hold an open hand for 2 seconds, then a fist for
2 seconds. Store the raw openness at each and remap so my fist is 0 and my
open hand is 1. For pinch, use hysteresis: pinched below 0.25, released above
0.35, so it never flickers at the edge.

The core of the smoothing is tiny, and it is worth understanding because you will tune it:

// alpha near 0.1 feels heavy and calm, near 0.5 feels fast and twitchy
function smooth(prev, next, alpha) {
  return prev + (next-prev) * alpha;
}
state.openness = smooth(state.openness, rawOpenness, 0.18);

Step 3: attach one visual

Pick one. Trying to do three at once is how these projects get messy.

A living graph

Nodes and links on a 2D canvas. Position moves the whole graph, openness scales it from a tight seed to a full bloom, rotation spins it. Keep labels upright. Cap the node count around 100 to start and draw glow with the canvas lighter composite mode instead of blur filters, which are expensive.

A scrubbed video

Render or generate a short clip, then map openness to playback time. Normal MP4 files only store a full frame every few seconds, so seeking stutters. Re-encode it so every frame is a keyframe:

ffmpeg -i clip.mp4 -an -c:v libx264 -g 1 -crf 18 -pix_fmt yuv420p -movflags +faststart clip-scrub.mp4

Keep the video paused and set currentTime only inside your animation loop, never in an event handler.

A 3D object

Use three.js. Rotation drives the object's rotation, pinch grabs it, openness controls an explode or scale effect. This is the one that looks best on video.

The prompt for any of them follows the same shape:

Using the signals from the debug page, add a [living graph / scrubbed video /
three.js object] on a dark full screen canvas. Map: position to [x], openness
to [y], rotation to [z]. Keep the debug view as a small corner overlay I can
toggle with the D key. When no hand is present, ease everything back to a
calm resting state over one second.

Make it hold up in front of people

• Light the hand well. Tracking quality drops fast in dim light or with a bright window behind you.
• Keep the fallback. The mouse mode saves you when a venue laptop has no camera permission.
• Respect reduced motion. If prefers-reduced-motion is set, slow the animation and skip big camera moves.
• Say what happens to the video. One line on the start screen, such as "your camera is processed on this device and never uploaded", makes people comfortable enough to try it.

If you want to go further into live installation work, TouchDesigner is the tool most people move to next, and there is an open source MediaPipe plugin for it. For a portfolio piece or a launch page, the browser version is usually enough.

Start with the debug page tonight and spend ten minutes just watching the numbers. Once openness and rotation feel stable in your own hand, every visual you attach will feel intentional instead of shaky.