FaceFit AR

Custom face detectors require GPU inference and tend to overfit lighting conditions. MediaPipe Face Mesh runs in WebAssembly on the CPU, returns 468 3D landmarks per face at 25 to 30 frames per second on a mid range laptop, and works across browsers without driver issues. Trading a few extra milliseconds for portability was the right call for a try on demo that has to work on whatever device a user already has.

Pose is computed by classical linear algebra rather than a PnP solver. Three landmark groups are used: left eye centre, right eye centre, and a forehead point. The eye to eye vector becomes the head's x axis. The forehead to eyes vector becomes an approximate up direction. Their cross product completes an orthonormal basis. This is fast and stable enough for accessory placement and avoids the convergence failures PnP solvers can hit when landmarks drift.

Depth uses MediaPipe's normalised z estimates. Scale is set by inter temple distance so the eyewear sizes correctly to the user's face regardless of how close they sit to the camera.

Webcam stream
    |
    v
MediaPipe Face Mesh   (468 landmarks, single face mode, confidence 0.5)
    |
    v
Pose estimator        (eye to eye + forehead, orthonormal basis)
    |
    v
Three.js renderer     (GLB overlay, transparent canvas over <video>)

Measured on a 2023 MacBook in Chrome:

Jitter is reduced by spatial averaging across landmark subsets (eyes, temples, nose bridge). Frames with confidence below threshold are rejected before reaching the renderer.

The frontend is a React app with three concerns kept separate: webcam streaming, perception (MediaPipe), and rendering (Three.js). Perception updates write to a React ref instead of state, so the 30 Hz landmark stream does not trigger re renders. The backend is a small Express service that serves the GLB product catalogue, deployed in a separate container so the perception pipeline can be reused with a different catalogue source later.

GLB models are recentered and bounding box normalised at load time so different eyewear geometries place consistently on the user's face.

Kalman or exponential smoothing filters upstream of the renderer would cut the remaining jitter further. Different accessory types (hats, earrings) would need different landmark subsets. The backend can swap to a database or external catalogue API without frontend changes.