Listen to phone audio
The agent can hear the iPhone. While the screen is mirrored, the board passes the phone’s sound to the app, which turns speech into text on the iPhone. The agent reads what it heard next to the screenshot, so it can answer a “Tap what you hear” exercise or follow a video.
Needs App 1.0.2 and board firmware v0.2.4 (TestFlight channel).
Turn it on
Section titled “Turn it on”- Open Settings → Listen to phone audio and switch it on. It is off by default.
- When iOS asks, allow speech recognition. If you said no earlier, turn it on in the iPhone’s Settings → ESP32 Agent.
- Pick the Second language, or leave Automatic (from the screen).
Languages
Section titled “Languages”The app listens for English plus one more language:
- Automatic (from the screen) (default): the app reads the screen and picks the language it shows, for example Japanese in a Japanese lesson. When the lesson changes language, it switches between two sounds, and a first phrase it heard too early is transcribed again.
- A fixed language: always listen for that one.
- None: English only.
Only languages with speech recognition on the iPhone itself can be picked; others are skipped.
What the agent gets
Section titled “What the agent gets”- The words, with the language, from the last seconds of sound. Not the sound itself.
- The agent decides when to play a sound, for example by tapping the speaker in an exercise. Nothing is played for it.
- Words can be misheard, especially short ones and similar-sounding ones.
Privacy
Section titled “Privacy”Speech is turned into text on the iPhone only; the sound never leaves the phone. The words go into the agent’s prompt like the screenshot does, so they reach the AI service you use. See the privacy policy.
Test it
Section titled “Test it”Settings → Developer → Audio test plays a short phrase through Screen Mirroring and shows what the app heard. Turn on Listen to phone audio first, and keep the board connected and mirroring on.
Handsy CLI
Section titled “Handsy CLI”With Share with the computer (CLI) on, Handsy CLI 0.1.3 or later reads the same words with esp32agent listen (MCP tool phone_audio). esp32agent listen mode local instead transcribes the raw audio on the computer with whisper.cpp; see the CLI guide.
If it hears nothing
Section titled “If it hears nothing”- Screen Mirroring must be on: the sound reaches the board only through AirPlay.
- Check the firmware: v0.2.4 or later; older boards don’t pass sound on.
- Run the Audio test and send diagnostics if it still fails.