ESP32 Agent

Setting up the ESP32 iPhone Agent comes down to the right board, the right cable and a few iPhone settings

Most setup problems with the ESP32 iPhone Agent trace back to hardware: the wrong ESP32 variant, a cable that only charges, or a phone that does not meet the minimums. This page covers what to buy, how to flash and connect it, and what to check when something does not work.

Choosing a board

The tested board is the Seeed Studio XIAO ESP32S3, standard version, with 8 MB flash and 8 MB PSRAM. It costs about $7. The Sense and Plus versions are not the tested board, so buy the standard one.

Board	Status
XIAO ESP32S3 (standard, 8 MB flash, 8 MB PSRAM)	Tested and supported
Other ESP32-S3 boards with native USB, at least 8 MB flash and 8 MB PSRAM	Should work, untested
ESP32-S3 boards using a CH340 or CP2102 serial chip	Not supported
ESP32-C3, ESP32-C6, ESP32-S2, original ESP32	Not supported

Phone and cable requirements

iPhone XS or newer, running iOS 18 or later.
Cellular data. The board's hotspot has no internet, so while the phone is joined to it, the model is reached over cellular.
A short USB-C to USB-C data cable, about 30 cm. Charge-only cables do not work, and they are the most common cause of a board that seems dead.
Lightning iPhones need a Lightning to USB Camera Adapter between the phone and the cable.

The phone powers the board. No battery or separate charger is involved.

Setup steps

Flash the firmware. Connect the board to a computer and open the install page in Chrome or Edge. The firmware installs from the browser. The iPhone cannot flash the board.
Get the app. Install the ESP32 Agent app from TestFlight, where it is offered as a public beta. If you build it yourself, that needs a Mac with Xcode once; it is not needed at run time.
Plug the board into the iPhone. Use the data cable, plus the Camera Adapter on Lightning models.
Turn on AssistiveTouch. The board acts as a mouse; with AssistiveTouch on, a click is a tap and a drag is a swipe.
Join the hotspot. The board opens its own Wi-Fi network. Connect the iPhone to it.
Start Screen Mirroring. Open Control Center, tap Screen Mirroring and pick ESP32 Agent.
Choose a brain. Either configure any OpenAI-compatible vision model in the app, or install Handsy CLI on a computer with npx -y esp32agent and drive the phone from there.

Troubleshooting by symptom

The board does not show up when flashing

Check that you are using Chrome or Edge on a computer, and that the cable carries data. Swap the cable before anything else. Confirm the board is an ESP32-S3 with native USB and not one with a CH340 or CP2102 chip.

"ESP32 Agent" is missing from Screen Mirroring

The board has to be powered by the phone and the phone joined to the board's hotspot. If the board is not getting power, suspect the cable or, on Lightning phones, the adapter.

Mirroring works but taps do nothing

AssistiveTouch is probably off. Without it, the iPhone does not turn mouse clicks into taps.

Taps land in slightly the wrong place

The USB pointer is relative. The app tracks it and re-homes it in a corner, and before precise taps it wiggles the pointer, finds it on screen and corrects the miss. Small misses are expected and corrected automatically.

The model cannot be reached

While joined to the board's hotspot the phone has no Wi-Fi internet. Make sure cellular data is on and allowed for the app.

The run stopped partway

If the board, hotspot or mirroring link is lost, the run pauses and resumes at the same step once the link returns. If the phone locked, mirroring stopped; unlock it and resume.

A screen appears black to the agent

iOS does not mirror passcode entry, Apple Pay or DRM video. This is an iOS rule, not a fault.

Text with accents or emoji will not type

Keyboard input is ASCII only.

Known performance limits

Bluetooth LE carries about 300–500 kbps, so the mirrored screen is 480×1040 at most 5 fps, and each step takes a few seconds, mostly model time. Landscape apps are not supported yet.

Checking what happened

Every run is saved on the phone and can be replayed step by step. To attach a run to a bug report, open Runs, open the run, tap ⋯ and choose Export trace. The source, MIT licensed apart from the GPL-3.0 AirPlay receiver in the firmware, is on GitHub.

Ready to build one

Flash your board from the install page. Board details are on the hardware docs, and more fixes are on the troubleshooting page.