Using the app
Tell Handsy what you want to do, then watch it use your iPhone. This guide takes you from a small board to your first task in the iPhone app.
Want to use an agent on your computer instead? See CLI & AI agents.
Watch it work
Section titled “Watch it work”See how the phone, board, and app work together in this walkthrough.
Get the app
Section titled “Get the app”Handsy AI is coming to the App Store; the download link will be available after launch.
Once the download entry is available:
- Open the App Store listing from this website on your iPhone.
- Tap Get and follow the App Store prompts to install.
- When installation finishes, tap Open to start the app.
What you need
Section titled “What you need”Need to choose a board or cable? The hardware & buying guide has the exact model, phone connections, and the same Amazon links as the homepage.
| Item | What to prepare |
|---|---|
| iPhone | iPhone XS or newer, running iOS 18 or later, with cellular data. USB-C models connect directly; Lightning models need a Lightning to USB Camera Adapter. |
| Board | Seeed Studio XIAO ESP32S3, the standard version with 8 MB flash and 8 MB PSRAM, including its Wi-Fi/Bluetooth antenna. The installer supports this version, rather than Sense or Plus. |
| Data cable | A USB cable that carries data, for the first firmware installation. A charge-only cable won’t work. |
| Computer, for the first setup | Desktop Chrome or Edge to prepare the board. The computer isn’t needed while a task runs. |
| AI service | Start with a free AI service available in the app. If you prefer your own service, use an OpenAI-compatible API that accepts images. |
During a task, your iPhone joins the board’s Wi-Fi hotspot for Screen Mirroring. That hotspot has no internet, so model calls use your cellular data.
1. Prepare the board
Section titled “1. Prepare the board”Do this once before plugging the board into your iPhone. This video shows the browser installation:
- Open the firmware installer in Chrome or Edge on a computer. An iPhone or Safari can’t perform this step.
- Connect the board with the data cable. Wait for Ready when you are., then click Connect board.
- Select the board in the browser’s USB list. If the page says it is restarting into install mode, click Connect board and select it again.
- Keep the board connected and the page open while installation runs.
- Wait until the page says Installed. Then unplug the board from the computer.
Can’t see the board? Try a different data cable, connect directly to the computer, and close other apps using the serial port. If it still doesn’t appear, hold BOOT while plugging it in, release BOOT, then try Connect board again.
If the firmware list doesn’t load, click Check again. If installation fails, unplug the board, reconnect it, and try Connect board again. See the FAQ if you’re still stuck.
Installation replaces the board’s existing firmware. It doesn’t change your iPhone or computer.
2. Connect it to your iPhone
Section titled “2. Connect it to your iPhone”Plug the prepared board into your iPhone and open the app, currently named ESP32 Agent. It walks you through four steps:
Check that the antenna is attached and keep the board close to your phone during setup. If pairing fails or the signal is weak, move it right beside the phone and check the antenna connection.
- Plug Me In — pair the board over Bluetooth.
- Let Me See the Screen — join the board’s Wi-Fi, then open Control Center → Screen Mirroring → ESP32 Agent.
- Let Me Use the Pointer — turn on Settings → Accessibility → Touch → AssistiveTouch. The app calibrates the pointer.
- Give Me a Brain — choose a free AI service available in the app to get started. To use your own service instead, enter its address and API key, then choose a vision model. Your key is stored in the iOS Keychain.
A free AI service is available now. We plan to offer a subscription AI service in the future for an easier way to get started. Read the privacy policy before choosing a service.
Keep your phone unlocked and in portrait. The board is powered by the phone.
3. Try a simple goal
Section titled “3. Try a simple goal”Enter “Open Settings and go to Wi-Fi”, then press Start.
Handsy looks at the screen, takes a step, and checks what changed. Follow its progress in the Dynamic Island or return to the app. You can stop the task from either place. If the preview is stuck on an old screen or actions keep repeating, stop and check the mirroring troubleshooting steps before retrying.
If it needs your help, press and hold the notification to respond. You may be asked to Continue, Skip step, Stop, or Take Over.
Start with a task you can watch. Screen Mirroring stops when the phone locks. Use portrait mode. For text entry, start with a short English phrase and check what appears on screen.
4. Keep what works
Section titled “4. Keep what works”Save a useful task to run it again with Siri, Shortcuts, widgets, or Control Center.
To review a run, open Runs and select it. Replay the steps with taps and swipes drawn on the screenshots. To export the record, choose ⋯ → Export trace.
For connection or pointer problems, see the FAQ. Before sending screenshots to an AI service, read the privacy policy.