Handsy AI
GitHub

ESP32 Agent

How ESP32 iPhone Agent works

ESP32 iPhone Agent is the open-source project behind Handsy AI. A Seeed Studio XIAO ESP32-S3 board plugs into a real iPhone so an AI can see the screen and tap, swipe, and type. No jailbreak. The iPhone app and the Screen Mirroring receiver show up as ESP32 Agent. The source is on GitHub, under the MIT license, except the firmware’s AirPlay receiver (GPL-3.0).

An AirPlay receiver, and a USB keyboard and mouse

The ESP32-S3 does both jobs. It opens its own Wi-Fi hotspot and appears in the iPhone’s Screen Mirroring list as ESP32 Agent. The phone mirrors over AirPlay. The board keeps the H.264 stream, without decoding it, and forwards it to the ESP32 Agent app over its own Wi-Fi (TCP). Bluetooth LE carries commands and status, and is the fallback for video if the Wi-Fi path isn’t available. Plugged into the phone’s USB port, the same board is a standard USB HID mouse and keyboard. Actions come back over Bluetooth, and the board carries them out over USB.

With AssistiveTouch on, a click is a tap and a drag is a swipe. The board’s hotspot has no internet. While the iPhone is joined to it, the model is reached over cellular data.

What decides the next step

Use the ESP32 Agent app on the same iPhone, or connect an agent on a computer with Handsy CLI. Either one looks at the screen and chooses the next tap, swipe, or keypress. The app decodes the stream and can send screenshots to an OpenAI-compatible vision model. It can keep working in the background; the Dynamic Island shows what it is doing. After each action it looks at the screen again.

Handsy CLI is the other path. An agent on your computer talks to the same board over Bluetooth, for both commands and video, asks for a screenshot, then sends taps, swipes, and typing. CLI & AI agents describes setup and what is published.

What you need

  • A Seeed Studio XIAO ESP32S3, the standard version with 8 MB flash and 8 MB PSRAM, not Sense or Plus, with the Wi-Fi/Bluetooth antenna attached. About $7. The browser installer supports this board.
  • An iPhone XS or newer on iOS 18 or later, with cellular data. USB-C phones plug in directly. Lightning phones need a Lightning to USB Camera Adapter. The phone powers the board.
  • A short USB-C data cable (about 30 cm / 1 ft). A charge-only cable will not work.

Keep going