Hand-Tracking Pen Plotter
A system that turns hand gestures from a webcam into real pen strokes on a serial-connected plotter, drawing on paper as the user sketches in the air. Built on a custom hanging polargraph machine with modified Marlin firmware, servo-controlled pen lift, and browser-based calibration tooling.
Why I built it
I built it to combine physical machine fabrication with playful interaction design — making drawing feel performative, surprising, and tangible.
How it works
The browser runs MediaPipe Hands for landmark tracking and feeds a live canvas. A Node.js / Express bridge remaps coordinates into plotter space and sends movement commands over SerialPort to the machine. The plotter itself runs on an ATmega-based Melzi board with a heavily modified Marlin v2.1 fork, with printing-specific code stripped out and polargraph kinematics, servo pen lift, and browser OpenCV calibration added.
Overview
Two layers make up the full system. The hardware layer is a custom polargraph — a wall-hanging machine driven by cable lengths rather than Cartesian rails — built by forking Marlin firmware, rewiring its motion model, and adding servo pen control. The interaction layer lives in the browser: MediaPipe tracks hand landmarks in real time, and a Node.js bridge translates that motion into G-code commands delivered over serial. The hard parts were not just tracking the hand or building the machine, but making the two systems feel directly connected — so that a gesture in the air becomes intentional ink on paper.
Process
Build the machine from a known firmware base
Marlin already solved many machine-control problems, so I forked it and removed the 3D-printing-specific code, then rewired the motion model around cable lengths and hanging geometry instead of Cartesian rails.
Track the hand and translate its motion
MediaPipe provided strong hand landmark detection. The real challenge was choosing the right point to track, smoothing it enough to feel drawable, and building a transform that preserved aspect ratio, centered output, and accounted for offset between camera and work area.
Close the loop between gesture and ink
Once interaction felt stable on screen, the last step was making the SerialPort bridge and firmware responsive enough that the physical drawing kept pace with the gesture — so the machine felt like an extension of the hand, not a delayed echo.
Outcome
The result is both a drawing machine and a firmware experiment: a weird and satisfying pipeline where gestures become drawing, and screen and machine feel coupled.