RPS Proof of Concept: Teaching a Computer to See Three Hand Shapes

Recognizing three hand shapes sounds like a machine learning problem. It is not. When MediaPipe gives you 21 landmarks per hand with coordinates for every fingertip and joint, the classification becomes a geometry problem.

This was the proof of concept that led to Gesture Duel. Single player, one hand, three gestures, and no game logic. Just the recognition pipeline to see if it would work at all.

The Classification

Each finger has a tip and a joint. If the fingertip y-coordinate is higher than the joint, the finger is extended. If it is lower, the finger is curled.

Paper is all five fingers extended. Rock is all five curled. Scissors is the index and middle extended while the ring and pinky stay curled. The thumb follows the same logic. Three conditions, one function, no training data. It runs in microseconds per frame.

What It Proved

The name says proof of concept because that is all it is. There is no opponent, no scoring, no rounds. The webcam feed shows your hand landmarks and prints the gesture name on screen. You make a shape. The computer tells you what it sees. That is the entire interaction.

This was enough to validate two things. The landmark coordinates were stable enough to classify against. And the fingertip-to-joint comparison was reliable across different hand sizes and camera distances. Both held under real conditions.

What I Would Change

The classification is rigid. There is no tolerance. A hand that is slightly open but not fully flat registers as unknown. A relaxed fist with a loose thumb fails. I would add thresholds to the comparisons instead of strict above-or-below checks.

It depends on lighting. The system works in good light and fails in dim conditions. MediaPipe landmark confidence drops and the classification becomes unreliable. There is no fallback.

The code is 81 lines and it does one thing. It proves the concept. The game logic came later in Gesture Duel. This one just watches your hand and tells you what shape you made. That was enough to know the rest was worth building.