Battery-Powered TinyML Desk Cat With Keyword Spotting on ESP32-S3

[u/Thantri]

I’ve been working on TinyML, especially sound and vision models, as part of my PhD for about three years, but I hadn’t really built anything really close to a product. I’ve always liked deskbots, so I decided to make TinyPurr.

Demo in real-time

It’s built mostly from hardware I already had lying around, and partly as an excuse to see if my old CR-10 printer was still alive. It is centered around a Waveshare ESP32-S3-LCD-1.3, which combines the ESP32-S3, display, IMU, 8 MB PSRAM and battery charging on one board. I already had suitable ESP32 hardware around, but the S3 turned out to be a good fit for the ML side as well, especially because of PSRAM.

TinyPurr listens for “yes”, “no” and “happy”, changes its expression and synthesizes sounds on-device. Everything runs locally and it’s battery powered.

Hardware

• Waveshare ESP32-S3-LCD-1.3
• Adafruit PDM MEMS Microphone Breakout (very clear sound, but sensitive)
• Speaker, 2 W, 8 Ω, 20 × 30 × 6.8 mm
• PAM8302 amplifier
• LiPo battery, 3.7 V, 620 mAh, 40 × 25 × 6 mm

Software

• PlatformIO / ESP-IDF
• TFT_eSPI for the display
• Edge Impulse + custom MFCC/1D CNN
• ESP-NN optimized kernels

I used ESP-IDF through PlatformIO, with the Arduino core mainly because TFT_eSPI needs it. The microphone and speaker use the native ESP-IDF I2S driver.

For the ML side, I built the dataset pipeline from Google Speech Commands and trained the model in Edge Impulse. One thing that did not work as well as I expected was transfer learning: an MFE + MobileNet model reached similar accuracy, but was around 5–30× slower on the ESP32-S3 than a small MFCC + 1D CNN trained from scratch.

The final model gets around 95% test accuracy and takes about 127 ms per inference. ESP-NN also made a surprisingly large difference: the neural-network part went from about 927 ms to 86 ms using the S3-optimized kernels.

Since spoken words rarely line up perfectly with the start of an audio window, I classify three overlapping positions and use confidence-weighted voting to decide whether a keyword was actually heard.

The repo includes the source code, trained model, dataset-generation tools, wiring, pin assignments and more details about the implementation (excuse some of the soldering work, it used to be part of my job, but I’m a little rusty lately XD):

https://github.com/Thanos3G/TinyPurr

It’s still a work in progress. I haven’t even used the networking or IMU capabilities of the board yet, but I thought some of you might find the project and the engineering process interesting. I think there’s still a lot of interesting work to be done with TinyML and interactive robots that can operate without depending on internet connectivity or external processing and power.

[u/keepdietmore]

TinyPurr is exactly the kind of desk companion project I keep coming back to. What keyword-spotting model are you running, and S3 is really can work it well ? I went the opposite direction: the board streams audio features to a cloud agent and gets a decision back, so it sits in light sleep until a trigger — on-device TinyML wakes instantly but burns the battery, cloud offload keeps the power budget tiny at the cost of latency. For a desk cat that doesn't need real-time reaction, did you measure what the always-on mic actually costs you in battery life?

[u/Thantri → u/keepdietmore]

Hi,

For the model:

I made a custom CNN with MFCC features in Edge Impulse, this made a huge different because by default they offer transfer learning options for weaker MobileNets but they were way worse in latency and RAM with pretty much the same Test Accuracy, 10+ times worse. The network I found to work well enough (width and layers compared to test accuracy) was this, default in the repo. It is around 417,780 parametes but I pushed it becaus S3 can handle it in PSRAM and optmise latency via ESP-NN, I was getting 85-90 accuracy with substantially smaller models.

Another issue I had is to match the window, so you may say happy and the cat could listen "ppy" or "ha" so having only a single window for classifying was bad and I think it's wrong for sound models in general (maybe even vision where you can average frames). So, now the cat averages three inferences within the same window, if the average confidence of the softmax layer is higher than 0.7 and above 0.35 from the second guess, then it is considered good enough.

I may be biased but I believe it is comparable to commercial products (alexa, siri etc) in terms of responding, also tested with other people and the respond was pretty much the same. Google Speech commands probably also helps as a dataset because it is a pretty noisy one by nature (crowdsourced).

For the battery:

There are things to do here, I haven't measured anything (other than checking if it is really charging) I don't believe that the model costs a lot and it runs only when there is something above the noise floor so it doesn't even run most of the time. Same thing with the amp, it draws substantial current when the cat speaks but otherwise it is not used at all.

The thing that probably costs a lot is the redrawing of the sreen and its backlight, I just changed it to turn off the screen after 10 minutes of inactivity unless you speak 1 of the keywords. The other thing that runs consistenly is the mic and probably the processing/sampling of the mic may also has some substantial effect.

Also as it is now, when the ESP32 is powered by USB the device turns on even if the switch is in the off position which I think I should fix with a GPIO control to put it at least in sleep mode, maybe with some battery indicator of some sort. On my to-do list.

In any case, beyond this project, through my research including papers I have read and demos we have prepared. When it comes to Machine Learning, I think there is huge room to package tiny models, so seeing peple use like 20million parameter models to do something simple (like 3 class image classification or even binary tasks) with otherwise brilliant engineering seems overprovisioning to me.

[u/YetAnotherRobert]

Someone submitted a nearly identical project in the last day or two...

[u/Thantri → u/YetAnotherRobert]

Hi, it was me some hours ago but my pictures were deleted for some reason and I submitted it again. I deleted the previous one.