Skip to content
Handsy AI · ESP32 iPhone Agent

ESP32 iPhone Agent: install, wiring and usage

ESP32 iPhone Agent is the open-source project behind Handsy AI. A Seeed Studio XIAO ESP32S3 board plugs into your iPhone and becomes its eyes and hands: the iPhone mirrors its screen to the board, and the board taps, swipes, and types as a standard USB mouse and keyboard. No jailbreak. The iPhone app and the Screen Mirroring receiver currently appear as ESP32 Agent.

This page sums up the whole setup and links to the full guides. The source code is on GitHub.

What you need

Section titled “What you need”
Item What to prepare
iPhone iPhone XS or newer, running iOS 18 or later, with cellular data. USB-C models connect directly; Lightning models need a Lightning to USB Camera Adapter.
Board Seeed Studio XIAO ESP32S3, the standard version with 8 MB flash and 8 MB PSRAM, including its Wi-Fi/Bluetooth antenna. The installer supports this version, rather than Sense or Plus.
Data cable A USB cable that carries data. A charge-only cable won’t work.
Computer, for the first setup Desktop Chrome or Edge to install the firmware. The computer isn’t needed while an app task runs.

Details and buying links: Hardware & buying.

1. Install the firmware from your browser

Section titled “1. Install the firmware from your browser”

Do this once, before plugging the board into your iPhone.

  1. Open the firmware installer at handsy.tech/install in Chrome or Edge on a computer. An iPhone or Safari can’t perform this step.
  2. Connect the board with the data cable and click Connect board, then select the board in the browser’s USB list.
  3. Keep the board connected and the page open until it says Installed. Then unplug the board.

Can’t see the board? Try a different data cable. If it still doesn’t appear, hold BOOT while plugging it in, release BOOT, then try Connect board again. Full steps and a video: Prepare the board.

2. Wiring: connect the board to your iPhone

Section titled “2. Wiring: connect the board to your iPhone”

Attach the included Wi-Fi/Bluetooth antenna to the board first. Without it, Bluetooth may not connect, or may work only with the board right beside your phone.

Your phone What to prepare
USB-C iPhone A short USB-C to USB-C data cable between the phone and board.
Lightning iPhone A Lightning to USB 3 Camera Adapter with a charger in its Lightning port, and a USB data connection to the board. See Lightning iPhone.

On USB-C iPhones the phone powers the board. Keep the cable short (about 30 cm / 1 ft) and make sure it supports data transfer. If the iPhone shows no pointer or keyboard from the board, try a different known-good data cable first.

3. How it sees and taps

Section titled “3. How it sees and taps”
  • Eyes: the board opens its own Wi-Fi and shows up in Screen Mirroring as ESP32 Agent. The iPhone mirrors its screen to the board over AirPlay. That hotspot has no internet, so model calls use your cellular data.
  • Hands: plugged into USB, the board is a plain mouse and keyboard. With AssistiveTouch on, a click is a tap and a drag is a swipe.
  • Brain: the board passes the screen to the iPhone app over its Wi-Fi, or to Handsy CLI on your computer over Bluetooth. The app or CLI decides the next step and sends it back over Bluetooth. If the app can’t use the Wi-Fi path, the picture falls back to Bluetooth, which is slower.

4. Use it with the app

Section titled “4. Use it with the app”

Plug the prepared board into your iPhone and open the ESP32 Agent app. It walks you through four steps:

  1. Plug Me In: pair the board over Bluetooth.
  2. Let Me See the Screen: join the board’s Wi-Fi, then open Control Center → Screen Mirroring → ESP32 Agent.
  3. Let Me Use the Pointer: turn on AssistiveTouch. Open Settings, search AssistiveTouch, turn on the switch at the top. It’s under Settings → Accessibility → Touch → AssistiveTouch. The app calibrates the pointer.
  4. Give Me a Brain: choose an AI service available in the app, or enter your own OpenAI-compatible service that accepts images.

Then try a simple goal such as “Open Settings and go to Wi-Fi” and press Start. Keep your phone unlocked and in portrait. Full guide with videos: Using the app.

4b. Or use it with your computer’s agent

Section titled “4b. Or use it with your computer’s agent”

Handsy CLI (esp32agent) connects over Bluetooth to the board plugged into your phone. Claude Code, Codex, or another agent on your computer can request a screenshot, tap, swipe, and type, then look again to check the result. Setup, demo, and availability: CLI & AI agents.

More help

Section titled “More help”

handsy.tech is the only official website of ESP32 iPhone Agent. Official links: GitHub · X.