Talking AI toy on ESP32-S3 with xiaozhi-esp32: on-board wake word, cloud LLM
A toy that talks back, built on open-source xiaozhi-esp32: the board hears the wake word, a cloud LLM answers. Plus kids' privacy and a mic light.
- The wake word is detected on the board with ESP-SR, but everything said after it goes to a server for speech-to-text, the LLM's answer and text-to-speech.
- The default firmware talks to the official xiaozhi.me server. Keeping audio at home means running your own server, which its authors say is not production-ready.
- A toy for children needs a visible listening light, network setup done by a parent, and an adult around while it is used.
- A child's voice is personal data. Thailand's PDPA requires a parent's consent for children under 10.
A toy you can talk to is now a weekend build: an ESP32-S3 board plus the open-source xiaozhi-esp32 project (MIT licence). The board listens for the wake word, records you and plays the reply. The “brain” is an LLM on a server. This lesson follows where the audio goes, how to get a prototype running, and what to settle before you hand the toy to a child.
Facts here come from the xiaozhi-esp32 README and Waveshare’s XiaoZhi AI guide for the ESP32-S3-AUDIO-Board, as of October 2026. The project moves fast, so check GitHub again before you start.
What runs on the board and what runs in the cloud
| Job | Where | Notes |
|---|---|---|
| Wake word | Board | ESP-SR (WakeNet), no internet needed |
| Record and compress speech | Board | Opus encoding |
| Send audio | Network | WebSocket or MQTT + UDP |
| Speech to text (ASR) | Server | |
| Answer (LLM) | Server / cloud | e.g. Qwen, DeepSeek |
| Text to speech (TTS) | Server | |
| Play audio, show a face | Board |
By default the firmware connects to the official xiaozhi.me server. If you want control, community servers run on your own machine, such as xiaozhi-esp32-server (Python). Its minimal install wants about 2 cores and 2 GB RAM with cloud APIs only, or 4 GB if FunASR runs locally. Its authors state plainly that it has not passed a network security assessment, should not be used in production, and needs your own protection if exposed to the public internet.
The project also has device-side MCP, which lets the LLM drive things on the board: volume, LEDs, a servo. That is how a toy can move when it talks, and it also means a cloud model can command hardware in a child’s room. Expose only safe actions.
Example board: ESP32-S3-AUDIO-Board
This ready-made board has everything a voice toy needs: ESP32-S3R8 (dual-core, 240 MHz), 16 MB flash, 8 MB PSRAM, two digital mics, ES8311 and ES7210 audio chips, a speaker amplifier, a ring of 7 RGB LEDs and a 3.7 V lithium battery connector with a charger. If you add a battery, read lithium batteries safely first.
Steps from the guide:
- Check the board is on XiaoZhi’s supported list.
- Flash a prebuilt firmware from the XiaoZhi or board maker’s GitHub, or build with ESP-IDF (
idf.py menuconfig, choose the board type). Current code needs ESP-IDF 6.0 or newer. - The board opens a hotspot named
Xiaozhi-xxxxxx. Join it from a phone; the setup page should pop up, or openhttp://192.168.4.1. Only 2.4 GHz Wi-Fi works. - Once online, the board shows a 6-digit code. Enter it in the xiaozhi.me console to bind the device to your account.
- Say “Hello XiaoZhi” or press BOOT, and start talking.
Worked example: how big is one sentence?
WakeNet takes 16 kHz, mono, 16-bit audio in 30 ms frames.
- Samples per frame = 16,000 × 0.030 = 480 samples
- Raw data rate = 16,000 × 16 bits = 256,000 bit/s = 32 KB/s
- A child speaks for 6 seconds: raw audio = 6 × 32 = 192 KB
- Compressed with Opus at 24 kbps (an assumed figure for comparison; the real setting may differ) = 6 × 24,000 ÷ 8 = 18 KB, about ten times smaller
Two things follow. Home Wi-Fi carries this easily. And every sentence after the wake word becomes a file that can be stored, forwarded and replayed, if the server chooses to keep it.
Children’s privacy
A child’s voice is personal data, and sentences often carry a name, a school or something about home.
- Thailand: under PDPA guidelines from the Personal Data Protection Committee, a child under 10 cannot consent alone; a parent must. If you build toys for others or sell them, ask someone who knows the law rather than guessing.
- United States: the FTC treats a child’s voice recording as personal information under COPPA. Its 2017 policy only relaxes enforcement when audio replaces typed words, is held briefly and then deleted, with the deletion policy stated clearly. A useful minimum bar to design to.
- Ask the server: the xiaozhi-esp32 README does not say how long audio is kept or where. Read the service’s own terms before binding a device. If you cannot find an answer, assume audio may be stored.
- Don’t collect what you don’t need: write the system prompt so the toy never asks for a real name, address, school or photo. Some boards have a camera; leave it disconnected unless it is needed.
A toy that listens must let an adult see every time it is listening, and know where the audio goes. If you can’t answer both, it isn’t ready for a child.
Cloud dependence: when the server goes, the toy goes quiet
The toy works only while the server does. If the service changes terms, shuts down or its LLM API changes, the toy stops answering. LLM answers are not fully predictable either: they can drift into topics unsuitable for the age or state wrong facts confidently.
To reduce the risk: run your own server at home, choose local ASR and TTS, write a prompt that limits topics to the child’s age, and don’t make the toy the only thing a child talks to at bedtime.
Mic light and parental supervision
- A clear status light: use the LED ring to show three states in different colours: waiting for the wake word, listening and sending, speaking. Both a child and an adult should be able to read it from across the room.
- Know its limits: a light controlled by firmware can be turned off by firmware. For stronger assurance, add a physical mic cut-off or power switch that is visible from outside.
- Parents do the setup: Wi-Fi, account binding and enabling MCP actions are adult jobs.
- Be there: especially with young children. Listen to what the model says; if an answer is not suitable, tighten the prompt or stop using it.
- Hardware: cap the maximum volume, since the speaker sits near small ears. Put a lithium battery in a case a child cannot open, and fix small parts firmly.
Common mistakes
- Thinking an on-board wake word means everything is on board. Hearing the sentence and answering all happen on the server.
- Using 5 GHz Wi-Fi. Setup only supports 2.4 GHz.
- Building with ESP-IDF 5.x. Current code needs 6.0 or newer.
- Exposing a self-hosted server to the internet without a password or firewall.
- No status light. Adults in the house cannot tell the toy is sending audio out.
For which ESP32 suits which job, see choosing an ESP32 board.
Sources
FAQ
Does the toy still talk without internet?
No. The board can spot the wake word by itself, but hearing the sentence, thinking of an answer and speaking it all go through the server. With the internet down it hears the wake word and cannot reply.
If I run my own server, does audio stay in the house?
It depends on which ASR, LLM and TTS you pick. xiaozhi-esp32-server has local options such as FunASR for speech recognition, but if the LLM or TTS is a cloud API, text or audio still goes to that provider.
Which board should I use?
The project supports ESP32, C3, C5, C6, S3 and P4, with more than 130 ready-made board folders. For a first build, pick an ESP32-S3 board with a mic and speaker that is on the supported list, so you don't wire the audio path yourself.
Circuits and code were tested on sample boards. Boards differ in pin layout, so check your board's pinout before you power anything.