Visioiner: Using Eye Tracking to Turn Attention Into Browser Control
Computer vision projects often stop at detection. They track a face, estimate a gaze vector, maybe draw landmarks on a frame. The interesting question for me was what happens when perception actually controls software.
Visioiner started from an annoyance with ads, but it became a project about stitching together systems that were never designed to work together. Eye tracking. Browser automation. Real time overlays. All inside one loop.
The Problem Was Not Eye Tracking
Detecting where someone is looking was only one part of it. The harder problem was converting noisy gaze estimates into something stable enough to drive behavior.
That is where calibration mattered. I used a five point mapping system, smoothing, and center pull adjustments because raw iris coordinates jitter. Without that, the whole idea falls apart.
- MediaPipe based gaze estimation mapped into screen coordinates
- Selenium driven YouTube ad detection through browser state
- Overlay and pause logic tied together with timing thresholds and cooldowns
The gaze overlay was a surprisingly useful piece. It was debugging infrastructure, but it also made the system feel tangible.
The Interesting Part Was The Glue
What made the project fun was the integration layer. Computer vision by itself was not the project. Connecting vision to browser behavior was.
Ad detection had to be responsive but not over poll. Pause logic had to avoid flickering states. Looking away had to mean something sustained, not one bad frame.
A lot of the engineering was thresholds, recovery logic, and deciding when not to act. That is usually where interactive systems become usable or annoying.
What Stayed Rough
This was clearly a focused prototype. The setup was fragile. Arc specific assumptions were hardcoded. Selector based ad detection could break if YouTube changed DOM behavior.
There were too many magic numbers. Some thresholds came from experimentation more than principled tuning. I would externalize and formalize that now.
I would also split the single file architecture apart much earlier. Computer vision, browser control, and UI should have been separate modules. Keeping them together helped move fast, but it made iteration harder.
What I Learned
One insight from this project was that perception systems are mostly uncertainty management. The model output is rarely the hard part. Deciding what confidence deserves action is harder.
I also learned browser automation gets interesting when it reacts to human state instead of scripted events.
Even small things like cooldown timing changed whether the system felt thoughtful or broken.
Why I Still Like This One
This was a weird project. That is partly why I like it. It took a small annoyance and turned it into an excuse to explore gaze estimation, automation, and interaction design at once.
Sometimes the best projects start as experiments and stay just useful enough to matter.