ASL Translator: Real-Time Sign Language Recognition with MediaPipe

ASL Translator was a computer vision project I built to understand the full pipeline from raw sensor input to classified output. The goal: real-time American Sign Language recognition from a standard webcam, with low enough latency to be usable in an actual conversation.

Motivation: most 'ASL translator' demos are frame-by-frame classifiers that work on controlled test images. I wanted something that worked on live video, under variable lighting, with different hand sizes and positions.

Why MediaPipe

MediaPipe's hand landmark detection gives you 21 3D coordinates per hand per frame — the joints of each finger, the palm center, the wrist. That's the right level of abstraction for ASL classification. You're not classifying pixels, you're classifying the geometric configuration of hand landmarks. That representation is robust to lighting variation, skin tone variation, and most background conditions.

Running landmark detection in real time on a consumer CPU is something MediaPipe handles efficiently. I didn't have to solve the 'how do I get 30fps hand tracking on a laptop' problem — I got to focus on what to do with the landmark data.

The Model Architecture

The classifier takes the 21 landmark coordinates (63 values: x, y, z per point) as input features. I normalized the coordinates relative to the wrist landmark so that hand position in the frame doesn't affect the classification — only the shape of the hand matters.

Trained on a dataset of labeled landmark sequences for the ASL alphabet and a set of common words. The model is a small feed-forward network — not deep learning for its own sake, but because the landmark features are compact and the classification task is tractable without a heavy architecture.

The Accuracy Challenges

Letters that look similar in static form — B and D, for example — are harder. The model gets them right most of the time. The errors cluster around letters with overlapping landmark configurations.

Signing speed is a bigger issue. Fast signers produce frames where the hand is between configurations, and the classifier sees an intermediate state it wasn't trained on. Temporal smoothing (averaging predictions over a small window) helps but introduces latency.

Lighting matters more than I expected. MediaPipe's landmark detection degrades in low light in ways that cascade directly into classifier performance.

What I Would Improve

Dynamic gesture recognition is the main limitation. ASL isn't just static hand shapes — many signs involve movement, and the current system only classifies static configurations. Extending to temporal sequences (an LSTM or transformer over landmark sequences across frames) is the obvious next step.

Two-hand support. Some ASL signs require both hands. The current system processes one hand at a time.

The gap between 'works in a demo' and 'works in the real world for real users' is large. It works in a demo. Closing that gap is a product problem as much as a technical one.