CLI & AI agents

Your agent already works on your computer. Give it a way to use your iPhone, too. Handsy CLI (esp32agent) connects over Bluetooth to the small board plugged into your phone. Your agent can request a screenshot, tap, swipe, and type, then look again to check the result.

Use Claude Code or Codex through MCP, or connect another agent that can run local commands or use a local stdio MCP server. The agent runs on your computer; the board receives the iPhone’s Screen Mirroring and acts as its mouse and keyboard.

Watch a real task

In this example, Claude Code uses an iPhone to open Chrome’s incognito mode, visit Google Maps, and search for coffee near East Broadway station in New York. It reads the actual screen and works through the task, rather than calling a Maps API.

Claude Code uses the CLI to operate a real iPhone, open Google Maps in Chrome, and find cafés near East Broadway station in New York. Some parts are sped up or cut.

Try the same task with your agent once connected:

Use esp32agent to open Chrome in incognito mode on my iPhone, visit the Google Maps website, and find coffee near East Broadway station in New York. Check a fresh screenshot after each action and stop when the map and café results are visible.

Install once, then hand it to your agent

The public CLI download is being prepared. The commands below describe the planned installation entry points and will work after the CLI release is published. The download scripts, npm package, and matching board firmware are not available from these entry points yet.

macOS or Linux

curl -fsSL https://handsy.tech/install.sh | sh

Windows PowerShell

irm https://handsy.tech/install.ps1 | iex

With npm

npm i -g esp32agent

The CLI needs Bluetooth on the computer. Screenshots also need ffmpeg; the installer can offer a package-manager command for your system. Follow its prompts and reopen your terminal if esp32agent is not found. The shell installer uses ~/.local/bin; make sure that directory is on your PATH.

1. Find and pair your board

Choose the supported board and attach its antenna using the hardware guide. This setup uses the upcoming firmware v0.2.2 and matching CLI, including the new wifi command. These releases are being prepared; do not assume the current browser installer supplies them yet.

You do not need to open the iPhone app. Plug the board into your iPhone, enable Bluetooth on your computer, and keep them nearby. Your computer can stay on its normal internet connection.

An agent with local terminal access can handle discovery and pairing. Give it this request:

Use your local terminal to set up esp32agent for my iPhone. Check that the compatible CLI and firmware v0.2.2 are available, discover my board, and pair if needed. Run esp32agent wifi, then guide me to join the board’s Wi-Fi and enable Screen Mirroring and AssistiveTouch on the phone. Verify a fresh screenshot and calibrate the pointer. Ask which board is mine if several are found. No iPhone app is required.

For manual setup, run:

esp32agent devices
esp32agent pair

pair discovers and pairs with a single nearby board itself; you do not need to pair it in the computer’s Bluetooth settings first. If more than one board is found, use the identifier returned by devices: esp32agent pair --device <id>.

Run the first pair within three minutes of powering the board. If that window has closed, unplug and reconnect it, then retry.

2. Connect Wi-Fi and mirror your screen

With firmware v0.2.2 and the matching CLI, read the board’s hotspot details after pairing:

esp32agent wifi

This upcoming command reads the hotspot name (ESP32Agent-…) and password over the paired, encrypted Bluetooth session. It does not change the board’s configuration. If the app is operating the board, wait until it is idle before retrying.

Then, on your iPhone:

1. Open Settings → Wi-Fi and join the hotspot using the name and password shown in your terminal.
2. Turn on Control Center → Screen Mirroring → ESP32 Agent.
3. Enable Settings → Accessibility → Touch → AssistiveTouch. Keep the pointer visible, and the phone unlocked and in portrait.

The phone uses the board’s Wi-Fi for mirroring; the computer continues using Bluetooth to reach the board. If you restart the board, reconnect the phone’s Wi-Fi and start Screen Mirroring again.

Check the current screen and calibrate:

esp32agent device info
esp32agent screenshot
esp32agent calibrate

The screenshot command saves a PNG on your computer and reports its filePath. Open that file and check that it shows the phone’s current screen. The CLI measures the pointer itself; the first pointer action also calibrates automatically if needed.

If the picture is stale, disconnect and restart Screen Mirroring. If pairing fails, check the antenna and move the computer closer. For an earlier version without wifi, see how to find the hotspot password.

3. Connect your agent

Complete the initial pairing in a terminal first, or ask an agent with local terminal access to do it using the request above. The MCP server provides phone controls; its tools do not perform the initial device discovery and pairing.

Claude Code

Register the local MCP server:

claude mcp add esp32agent -- esp32agent mcp

You can also install the CLI’s Claude skill:

esp32agent skill install

This installs the skill under ~/.claude/skills/esp32agent/. It gives Claude the phone workflow and command guidance; MCP provides the tools.

Codex

Register the same MCP server with Codex:

codex mcp add esp32agent -- esp32agent mcp

Another agent

Use esp32agent mcp as a local stdio MCP server, or let your agent call the CLI directly. It needs access to the computer’s Bluetooth and the saved screenshots. A remote agent without access to this computer cannot reach the board by itself.

For a setup request you can paste into an agent, use the homepage’s CLI entry. The agent setup guide contains the installation and verification steps in plain Markdown.

4. Give it a small first task

Try this request in your agent:

Use esp32agent to take a screenshot of my iPhone. Open Settings and go to Wi-Fi. Take a fresh screenshot after each action to check the result. Stop when the Wi-Fi page is visible.

The agent should look → take one action → look again. Watch the first task and interrupt your agent if it goes off track. For a longer task, describe the destination and what counts as finished.

The CLI provides the controls; your agent supplies the reasoning and AI service. The free AI service in the phone app is a separate way to use Handsy.

Useful commands

These are examples of individual actions. Choose coordinates from a fresh screenshot before running a tap or swipe; coordinates are pixels in the original screenshot, not percentages.

esp32agent screenshot
esp32agent io tap 200,400
esp32agent io swipe 200,800,200,300
esp32agent io text 'hello'
esp32agent io keys enter
esp32agent io button HOME

Chinese typing uses the phone’s Pinyin keyboard and requires selecting the intended candidate. Enable that keyboard first and have the agent check the text on screen.

Switch between the app and CLI

Use them in turns. An active app operation has priority and can interrupt the CLI connection. Stop the app task before starting a CLI task; if the CLI reports that the board is busy, wait before retrying. If the picture stops updating after a switch, restart Screen Mirroring and check a fresh screenshot.

CLI history is stored on your computer. Inspect the latest run with esp32agent history show latest. Intent files (.esp32intent) can be exported and moved between the computer and app with Files or AirDrop; this is a file handoff, rather than automatic history synchronization.

Screenshots may be sent to the model used by your computer agent. Check that agent’s permissions and provider’s data policy, as well as Handsy’s privacy policy, before using personal accounts or screens.

Need help with mirroring, the antenna, or the pointer? See the FAQ. Use only devices and accounts you own or are authorized to control; see the disclaimer.