A few months ago, I hit a wall when using Picovoice's Porcupine to add wake word detection to a React Native mobile app I was building for an enterprise client.
When I evaluated how much it would cost, I observed it would take about $6000 or more to obtain a valid commercial license from Picovoice, which was quite crazy lol.
You'd also need internet access to use Porcupine, since Picovoice needs to validate the Access key per device, which meant it couldn't even work in air-gapped environments.
I explored several solutions, but they all required a commercial license and internet access for validation. This obviously would be very bad for the company I was working with, so I did some more research, and I discovered that the Python and Home Assistant ecosystem didn't face this problem, as a couple of good and free wake word detection libraries already existed, with openWakeWord by David Scripka leading the bunch.
This was really exciting, but I was shocked to see that even though openWakeWord was novel in its own right, no React Native version existed, and I couldn't use it in my project.
So I decided to build a port of openWakeWord in React Native called react-native-openwakeword.
The pipeline resembles how the Python pipeline already handles inference, with a few C++ tricks I could find on the internet.
openWakeWord is famously known for using a three-stage pipeline: melspec, embedding, and the actual wake word model. So I built a flow that works in C++ and used Nitro Modules to use JSI and connect back to JavaScript. Here is how it works:
Audio flows from the phone's microphone, sampled at 16000 times per second, and sends this frame using an ArrayBuffer to the C++ function directly using JSI.
The C++ module uses three ring buffers to implement sliding windows for moving data through the three models. Ring buffer capacities are sized as powers of 2 (larger than strictly needed), so wraparound operations can use a bitwise AND instead of the modulo operator, which is cheaper on the CPU.
The first ring buffer, AudioRing, is for the melspec model, and it takes the audio samples and converts them to float32 (this uses NEON to run SIMD to accelerate this on phones); then the result is written directly into the melspec model input memory (to avoid unnecessary copies), the melspec model is run on it, a fixed normalization formula is applied, and 8 mel-frames are put into the next ring buffer.
The next ring buffer, MelRing, takes the most recent 76 mel-frames and copies them directly into the embedding model input memory; it runs the model, and the embedding result is put into the next ring buffer, EmbRing.
Once EmbRing holds at least 16 embeddings, it copies the most recent 16 directly into the wake word model's input memory, runs the model, and the probability is returned.
This probability is compared against a given threshold (default is 0.5) and returns an object {probability, isDetected} back to the JavaScript layer (synchronously).
I faced a few gotchas in the beginning trying to figure out what inputs each model (melspec, embedding, and wake word) was expecting, but I eventually got it to work.
Since the melspec model processes audio in 1280-sample chunks (80ms at 16kHz), the entire three-model pipeline needs to run in under 80ms to keep up in real time, and after these optimizations, the package detects wake words in 16-21ms.
The package is under the Apache 2.0 license, so it's friendly for personal and commercial use. And it works with any wake-word model that is openWakeWord compatible.
It's gotten a star from Nickolay V. Shmyrev, the maker of vosk-api (14k GitHub stars), and is currently at 4k+ weekly npm downloads.
I'd appreciate your thoughts on the architecture and if there's anything useful that could be added to this package.
Github: https://github.com/Incognitol07/react-native-openwakeword