Grabby: Turning Push-Ups Into Currency, a Cursor, and a Robot

Most fitness gamification ends at a badge or a streak. I wanted something more literal. Physical effort should produce a tangible output, not just a number in a database.

At Scrapyard Hacks Toronto, my team built Grabby. A push-up counted by a webcam creates a virtual click. That click can be spent through a voice command or a hand gesture. An Arduino with two servos points a laser at whatever the cursor is doing. The loop is closed.

The Pipeline

The push-up counter uses MediaPipe Pose to track the elbow angle. Extended past 160 degrees is the top. Below 90 is the bottom. A cooldown prevents double counting. Each rep writes one click to a text file.

The voice module reads that balance and listens for commands. Say "click" and pyautogui fires a click. The balance decrements. The hand gesture module tracks the index finger MCP landmark through MediaPipe Hands and moves the cursor to match. A moving average deque smooths the jitter that raw hand tracking produces.

The three systems share one text file. No locks. No atomic writes. At hackathon scale, it worked.

The Robot

The Arduino receives screen coordinates as CSV over serial. It maps the 1920x1080 range into servo angles and drives pan and tilt to point a laser at the cursor position. The laser moves live. It is visible on the wall, on the monitor bezel, wherever the cursor goes.

That part turned the project from a screen demo into something physical. Judges could see the cursor leave the monitor and appear as a dot on the wall. The abstraction gap between a push-up and a mouse click became concrete.

What Did Not Survive the Weekend

The screen resolution is hardcoded. The serial port path is hardcoded. The push-up detection assumes the left arm. The click balance file has no concurrency protection, so running voice and gesture modules at the same time could race. The voice recognizer recurses on error and can stack overflow if the microphone is unavailable.

These are hackathon limitations. They are real and I would fix them in a second version. Resolution detection, serial port discovery, bilateral arm tracking, and a proper shared state model would take an afternoon.

I would also clean up the dead code. temp.py exists but is unused. It was an alternative color tracking approach we did not integrate.

Why It Worked

The project won fourth place out of 60 teams. Not because any single module was impressive. The push-up counter was straightforward. The voice clicker was simple. The Arduino robot was two servos and a laser.

What worked was the loop. The physical act of dropping to the floor and doing push-ups affected what happened on screen. And what happened on screen became a physical dot on the wall. That kind of integration is rare at hackathons. Most teams build one thing well. We built three things that talked to each other, and the whole was more interesting than the parts.

A hackathon project does not need to be production ready. It needs to close the loop.