ESP32 Agent
You can let an AI control your iPhone with a $7 board plugged into its USB port
iOS does not offer apps a way to tap and swipe inside other apps. Handsy AI (the ESP32 iPhone Agent) works around that from the outside, the same way a person would: it watches the screen and moves a pointer. A small ESP32-S3 board plugged into the iPhone acts as a USB mouse and keyboard, while the iPhone mirrors its screen to that board. An AI vision model looks at the screen, decides what to do, and the board performs the tap. No jailbreak and no developer account are needed for daily use.
This page walks through the whole path, from buying the board to watching a run finish.
What you need
| Item | Requirement |
|---|---|
| Board | Seeed Studio XIAO ESP32S3, standard version (8 MB flash, 8 MB PSRAM), about $7 |
| iPhone | iPhone XS or newer, iOS 18 or later, with cellular data |
| Cable | Short USB-C to USB-C data cable, about 30 cm. Charge-only cables do not work |
| Lightning iPhones | Add a Lightning to USB Camera Adapter |
| Computer | Chrome or Edge to install the firmware |
| Build from source (optional) | A Mac with Xcode, only if you build the app yourself instead of using TestFlight; not needed at run time |
The board draws power from the phone, so there is no battery or wall adapter to manage. Do not buy the Sense or Plus variants of the XIAO; get the standard one.
How the AI actually sees and touches the screen
There are three links between the phone and the board, and each one has one job.
The screen goes out over AirPlay
The board opens its own Wi-Fi hotspot and shows up in the iPhone's Screen Mirroring list as ESP32 Agent. When you pick it, the phone mirrors its screen to the board over AirPlay. The board does not decode the video. It keeps the H.264 stream as is and forwards it to the ESP32 Agent app over its own Wi-Fi (TCP).
Commands travel over Bluetooth LE
Bluetooth LE carries commands and status between the app and the board. It is also the fallback path for video if the Wi-Fi link is not available.
Touches go in over USB
On the phone's USB port, the board is a standard USB HID mouse and keyboard. With AssistiveTouch turned on, a click becomes a tap and a drag becomes a swipe. The board can also type text and press keys.
Because the hotspot has no internet, the phone reaches the AI model over cellular data while it is joined to the board's Wi-Fi. That is why cellular data is on the requirements list.
Setting it up, step by step
- Plug the XIAO ESP32S3 into a computer and open the install page in Chrome or Edge. Flash the firmware from the browser. The iPhone cannot do this step.
- Install the ESP32 Agent app on the iPhone. It is distributed through TestFlight as a public beta.
- Connect the board to the iPhone with the short USB-C data cable (or the Camera Adapter on a Lightning phone). The phone powers it.
- Turn on AssistiveTouch in the iPhone's accessibility settings so mouse clicks register as taps.
- Join the board's Wi-Fi hotspot, then open Control Center, tap Screen Mirroring and choose ESP32 Agent.
- In the app, enter the address and key for any OpenAI-compatible vision model you want to use.
- Type a task in plain language, such as opening an app and finding a setting, and start the run.
What happens during a run
The app decodes the mirrored stream and sends screenshots to the model. It runs a planner and an actor. The planner works out the next step; the actor turns it into one of a small set of actions: tap, swipe, draw, type, press keys, go home, or finish.
A mouse pointer is relative, not absolute, so the app has to know where it is. It tracks the pointer and re-homes it in a corner. Before a precise tap it wiggles the pointer, finds it on screen, and corrects any miss. After each action it waits for the screen to settle and tells the model whether anything changed, so the model is not guessing.
If the board, the hotspot, or the mirroring link drops, the run pauses and resumes at the same step instead of starting over.
Every run is kept on the phone. You can replay it step by step to see what the model saw and what it chose. To share one, go to Runs, open the run, tap ⋯ and choose Export trace.
Phone app or a computer: two ways to give it a brain
| ESP32 Agent app | Handsy CLI on a computer | |
|---|---|---|
| Where the agent runs | On the iPhone | On your computer |
| Video path | Board Wi-Fi (TCP), BLE fallback | Bluetooth |
| Model | Any OpenAI-compatible vision model | Whatever agent you run, via MCP or the CLI |
| Install | TestFlight | npx -y esp32agent |
The app is the simplest path if you want the phone to work on its own. The CLI suits people who already run an agent on a desktop and want to give it the phone as a tool. Control an iPhone from Claude Code covers that route in detail.
Limits you should know before you start
- Bluetooth LE carries about 300–500 kbps, so the mirrored screen is small (480×1040, at most 5 fps). Each step takes a few seconds, and most of that time is the model thinking.
- iOS does not mirror passcode entry, Apple Pay or DRM video, so the AI cannot see those screens.
- Mirroring stops when the phone locks.
- Keyboard input is ASCII only.
- Landscape apps are not supported yet.
These limits make it a fit for multi-step tasks in ordinary portrait apps, not for games or anything that needs fast reactions.
Open source
The project is on GitHub under the MIT license. The firmware's AirPlay receiver is the one exception and is GPL-3.0.
Get started
Order a XIAO ESP32S3, find a data cable, and follow the install guide to flash the board and get the app. For a slower walk through each screen, read the getting started docs.
Related: iPhone MCP server · ESP32 iPhone agent setup · Control an iPhone from Claude Code · How it works