Handsy AI for iPhone

An AI that uses your iPhone. With its own hands.

Plug a $7 ESP32 board into your phone. It sees the screen and taps through USB. Give the iPhone app a goal, or connect your computer’s agent with Handsy CLI. No jailbreak.

Get started GitHub Watch the demo

App Store coming soon · CLI release in preparation

Task Set an alarm for 7:00 AM that repeats every day

Reading the screen…1/12

Hi, I'm the agent. Poke me.

Tap · double-tap · hold · drag

Get started

Your iPhone. Two ways in.

Use Handsy on your phone, or put your computer’s agent in control of the same board.

A goal, and a pair of hands.

The app walks you through connecting the board, seeing the screen, and starting your first task.

Coming to the App Store. The download link will be added after launch.

Prepare the board once in your browser.
Follow the four setup steps on your iPhone.
Give Handsy a goal. Watch it work, or step in.

Open the setup guide Choose a board and cable

Give your agent an iPhone.

Handsy CLI connects your computer to the board over Bluetooth. Your agent can see screenshots, tap, swipe, and type on your real phone.

CLI release in preparation.

The commands below are for the upcoming release. Ask your agent to check availability before installing.

Your agent finds and pairs the board over Bluetooth. With the upcoming firmware v0.2.2 and matching CLI, esp32agent wifi reads the hotspot details. Join it in iPhone Settings and turn on Screen Mirroring and AssistiveTouch; no phone app is required. Phone connection steps

1. Install the CLI

curl -fsSL https://handsy.tech/install.sh | sh

irm https://handsy.tech/install.ps1 | iex

2. Hand setup to your agent

Copy this request into an agent with local terminal access. It will check release availability, then discover and pair your board and verify a screenshot.

Read https://handsy.tech/cli-guide.md and help me set up Handsy CLI using your local terminal. Check availability of the matching CLI and firmware v0.2.2, install if needed, discover and pair my board, and run esp32agent wifi. Guide me through joining the hotspot in iPhone Settings and enabling Screen Mirroring and AssistiveTouch. Then verify a fresh screenshot and calibrate the pointer. Ask which board is mine if several are found. No iPhone app is required.

Connect an agent yourself

After installing the CLI, choose your client to see its MCP command.

claude mcp add esp32agent -- esp32agent mcp

codex mcp add esp32agent -- esp32agent mcp

Claude Code uses the CLI to operate a real iPhone, open Google Maps in Chrome, and find cafés near East Broadway station in New York. Some parts are sped up or cut.

How it works

One board. Two ways to give it a brain.

The board is the eyes and hands. Use the app on your iPhone, or your agent on a computer.

01 · Eyes

It sees your screen

The board opens its own Wi-Fi and shows up in Screen Mirroring. It passes the AirPlay stream over Bluetooth to the app or CLI.

02 · Hands

It taps like you do

Plugged into USB-C, the board is a plain mouse and keyboard. With AssistiveTouch on, a click is a tap and a drag is a swipe.

03 · Brain

Your app or your agent

Let the iPhone app plan the next step, or connect your computer’s agent through the CLI to see screenshots and act on the phone.

iPhoneScreen Mirroring

AirPlay videotaps over USB

ESP32-S3$7 XIAO board

Bluetoothnext step

App or CLIyour AI → next step

On X and in the press

See it run, and who’s talking about it.

Henry@iamhenryyuan

A $7 ESP32 lets an AI agent control a stock iPhone. THAT'S ALL. … The ESP32 takes the iPhone's AirPlay screen and taps back over USB. We also built a very cute agent app that works together with the ESP32 to automate any iPhone. Show more

Henry@iamhenryyuan

A $7 ESP32 lets an AI agent control a stock iPhone. No Mac, no jailbreak, no developer mode. Just a Linux machine. No cloud APIs for this demo: Qwen3.8-Flash-Next runs locally on 2× DGX Spark (TP=2), everything stays on my LAN. Open source 👇

Seeed Studio@seeedstudio

An AI that uses a real iPhone, with a $7 board as its eyes and hands 📱 Henry Yuan's ESP32 iPhone Agent runs on our XIAO ESP32S3: the phone mirrors its screen to the board, and the board taps back as a USB mouse. No jailbreak.

Hackadayhackaday.com

AI Controls iPhone Via ESP32

Do you want to get an AI agent to do something to an iPhone? You could try and jailbreak the device and do some heavy software hacking, or… you could grab a microcontroller off the shelf.

What you need

Less than a lunch.

Board

XIAO ESP32S3, about $7

Choose the standard version with 8 MB flash and 8 MB PSRAM for the browser installer. Attach the included Wi-Fi/Bluetooth antenna.

Buy on Amazon ↗

Phone

Any iPhone on iOS 18

iPhone XS or newer. USB-C models plug straight in; Lightning models use a USB camera adapter. Use the phone app or connect a computer agent through the CLI.

To connect the board, a short USB-C to USB-C data cable.