AI summary · arXiv: 2610.38739v1 · eess.AS
Streaming Speech Recognition on a Microcontroller: a 65.7M-Parameter Transducer in 2 and 4 Bits on the ESP32-P4
The essentials
Training-free mixed 2/4-bit quantization enables a 65.7-million-parameter streaming speech recognizer to run faster than real time on an ESP32-P4, with modest but dataset-dependent accuracy losses.
Question
Can an open-vocabulary streaming speech recognizer with tens of millions of parameters fit and run faster than real time on a microcontroller without a neural accelerator? The paper investigates the compression, integer arithmetic and implementation choices needed without retraining the deployed model.
Method
The author quantized Kroko’s 290 encoder-path matrices using sensitivity-based 2/4-bit allocation, Hadamard-rotated groups, sequential correction, GPTQ and group-norm search, calibrated on 512 FLEURS training clips totaling 87 minutes. A C engine uses int16 intermediate activations, int8 matrix-product inputs and hand-written SIMD kernels on two 400 MHz ESP32-P4 cores with 32 MB PSRAM. Accuracy was evaluated through a bit-exact PC build on complete LibriSpeech test sets, 647 FLEURS clips and 1000 Common Voice clips, with quantization ablations and on-device timing experiments.
Findings
-
Result
The deployed weights shrink from 71 MB to 26.65 MB, and the live application processes each 1.28-second audio chunk in approximately 0.90 seconds.
See the passage in the paper
The released 71 MB of int8 weights become a 26.65 MB file.
-
Result
With beam search of width 4, LibriSpeech WER is 4.34% on test-clean and 10.26% on test-other, compared with 3.94% and 9.17% for released Kroko.
See the passage in the paper
On LibriSpeech test-clean [25] it reaches 4.34 % WER and on test-other 10.26 %, with a beam of 4, against 3.94 % and 9.17 % for the released model run with sherpa-onnx.
-
Result
In the test-clean ablation, GPTQ accounts for most of the improvement over nearest-level rounding under the mixed-bit allocation.
See the passage in the paper
Rounding to the nearest level on the 25 MB plan nearly doubles the WER, to 8.83 %. GPTQ alone does most of the repair (4.75 %).
-
Authors’ interpretation
Simulator comparisons attribute the accuracy degradation to weight quantization rather than integer arithmetic.
See the passage in the paper
The whole gap comes from the weights. The integer arithmetic adds nothing here either.
-
Result
The beam-4 WER gap increases to 2.26 percentage points on FLEURS and 1.09 points on Common Voice, with paired-bootstrap intervals excluding zero.
See the passage in the paper
Both models make three to five times more errors here than on LibriSpeech, and the gap between them grows: 2.26 points on FLEURS and 1.09 on Common Voice at beam 4, against 0.40 on test-clean, and the bootstrap intervals exclude zero on both sets.
Limitations
- Energy and power consumption were not measured.
See the passage in the paper
We did not measure energy or power; we had no instrument for it.
- Evaluation covers English read or prompted speech, leaving noisy, far-field and conversational performance unestablished.
See the passage in the paper
The system is English only. The test sets are read or prompted speech: LibriSpeech, FLEURS, Common Voice, and text-to-speech clips for the phrases.
- The quantization method was developed on a single model, and transfer of its settings to other models remains untested.
See the passage in the paper
Finally, the method was developed on one model; we have not shown that the same settings transfer to other models.
Why it matters
The paper demonstrates that training-free compression and carefully implemented integer inference can make a large streaming transducer practical on a microcontroller without a neural accelerator. The authors identify weight quantization, rather than integer arithmetic, as the remaining accuracy bottleneck.
Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
O\'ido: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural AcceleratorDaniel Varela Sánchez, Guillermo Martín-Coello Juárez · 2026 · cited 1 time · Not summarized yet
O’ido runs open-vocabulary English speech recognition on a microcontroller without a neural accelerator, using an int8 Conformer on the ESP32-S3.
Abstract
Speech recognition on low-cost microcontrollers is, in practice, limited to keyword spotting or a fixed list of commands. We present O\'ido, an inference engine that runs open-vocabulary English speech recognition entirely on an ESP32-S3 (240 MHz dual-core Xtensa LX7, 512 KB of SRAM, 8 MB of PSRAM, 16 MB of flash, no neural accelerator). O\'ido executes a 13M-parameter Conformer-CTC model with int8 weights, per-frame dynamic int8 activations and int8 attention on the chip's SIMD instructions, with a schedule designed around flash and PSRAM bandwidth. On LibriSpeech test-clean/test-other the on-device arithmetic reaches 3.70%/8.23% word error rate (WER), within 0.12 points of full precision, …
-
Realizing Low-Precision Storage and Sparse Execution in a Folded FPGA Speech EnhancerClément Laroche · 2026 · Not summarized yet
Realizing Low-Precision Storage and Sparse Execution compresses a streaming speech-enhancement model for FPGA deployment, rather than a speech recognizer.
Abstract
Hardware-aware training often optimizes FPGA resources, latency, or bit operations, but folding changes what compression saves: shared arithmetic decouples precision, weight storage, and scheduled work. We derive training objectives from the measured costs of a folded streaming speech-enhancement accelerator. Mixed-precision weights reduce on-chip storage; structured channel sparsity removes entire operations from the engine schedule. We deploy a bit-exact implementation on the Kria KV260. The best sparse model eliminates 47.3% of convolution MACs, cutting frame time by 45% (1.86 to 1.03 ms) and dynamic energy by 60% (83 to 33 $\mu$J/frame) against a storage-priced model measured in the same…
-
How Far Can the Gemma 4 Speech Encoder Go Without Changing Its Weights? Saudi and Gulf Arabic Speech Recognition with a Frozen and an Adapter-Augmented EncoderAmr Abdelkhalik · 2026 · Not summarized yet
How Far Can the Gemma 4 Speech Encoder Go studies Arabic recognition with a frozen pretrained encoder, not low-bit microcontroller inference.
Abstract
We study the 305M-parameter audio encoder shared by Gemma 4 and EmbeddingGemma 2 (bit-identical to that of Gemma 4 E2B except for the final audio-embedding projection) for Saudi and Gulf Arabic speech recognition, without changing any of its pretrained weights; the adapters of our adapted system are additive and kept separate. A frozen system trains only a 22.6M-parameter CTC reader over a learned mix of four intermediate encoder layers and the encoder output; an adapted system adds 3.1M low-rank adapter parameters, a 41.5M-parameter hybrid CTC/attention reader and joint decoding with a sub-word language model. Against Cohere Transcribe Arabic (2.07B parameters), re-run on identical audio, t…
-
Edge-ASR: Towards Low-Bit Quantization of Automatic Speech Recognition ModelsChen Feng, Yicheng Lin, Shaojie Zhuo et al. · 2025 · Sound · preprint, not peer reviewed · cited 12 times · AI summary available
Edge-ASR evaluates post-training low-bit quantization methods for automatic speech recognition models without retraining.
Abstract
Recent advances in Automatic Speech Recognition (ASR) have demonstrated remarkable accuracy and robustness in diverse audio applications, such as live transcription and voice command processing. However, deploying these models on resource-constrained edge devices (e.g., IoT device, wearables) still presents substantial challenges due to strict limits on memory, compute and power. Quantization, particularly Post-Training Quantization (PTQ), offers an effective way to reduce model size and inference cost without retraining. Despite its importance, the performance implications of various advanced quantization methods and bit-width configurations on ASR models remain unclear. In this work, we pr…
-
Real-Time Music Source Separation on a Low-Power Audio DSPJia-Nan Li, Li Liu, K. Malsky et al. · 2026 · Not summarized yet
Real-Time Music Source Separation tests real-time audio-model deployment on a constrained DSP, focusing on source separation rather than recognition.
Abstract
Real-time music source separation is validated on desktop CPUs and GPUs. Does any published system fit the embedded audio hardware it targets? On a commercial audio DSP (2 MB SRAM, 2.07 GMAC/s measured), none does, and the constraints eliminate different models: memory rules out the 16-51 M parameter TasNet/X-UMX family, per-frame compute rules out RT-STT, needing 5.5x the available MAC rate. Parameter count predicts neither: weight reuse spans 1x to 345x. We then build one that fits. Training on continuous rather than block-padded convolution context proves essential: a model scoring 3.93 dB block-wise otherwise collapses to silence within 2 s frame-by-frame. A gated complex FIR deep filter…
-
Paradee: Distilling Kokoro-82M into an 8M-Parameter Single-Voice Text-to-Speech ModelSahil Mahendrakar · 2026 · Not summarized yet
Paradee compresses a text-to-speech model through distillation and quantization, rather than deploying a speech recognizer.
Abstract
We distill Kokoro-82M, a widely used open text-to-speech model with 54 voices, into Paradee, an 8.07M-parameter model that speaks one of them. Paradee keeps Kokoro's architecture with much narrower layers, and each of its two halves is trained separately against the frozen teacher. It has 10x fewer parameters and needs 15x less compute. We first synthesize a corpus with the teacher and keep its durations, pitch, energy and phoneme features. We then train a small text side to predict these values, and a small decoder to turn the teacher's saved values into the teacher's audio, first with spectral losses and then adversarially. Finally, we connect the two halves and quantize the weights to int…
-
Beyond Model Size: Redesigning LiSenNet for embedded speech enhancementClément Laroche, R. Olsson · 2026 · Not summarized yet
Beyond Model Size redesigns a speech-enhancement network for a microcontroller NPU, whereas this paper targets recognition without a neural accelerator.
Abstract
Deploying real-time speech enhancement on resource-constrained devices requires meeting strict latency, memory, and energy constraints. Microcontroller NPUs can accelerate neural inference under these constraints, but only through a restricted set of operators in static, integer-quantized graphs. Recent speech-enhancement networks have reduced parameter counts and MACs to levels nominally suitable for microcontrollers, but their operators and execution patterns often remain incompatible with restricted NPUs. We address this gap by redesigning LiSenNet, a 37k parameter sub-band dual-path model, for the STM32N6570-DK Neural-ART accelerator. We replace its recurrent bottleneck with convolutiona…
-
2-bit Conformer quantization for automatic speech recognitionOleg Rybakov, Phoenix Meadowlark, Shaojin Ding et al. · 2023 · Interspeech · cited 23 times · Not summarized yet
2-bit Conformer quantization studies two-bit weight quantization for ASR, though it evaluates Conformer models rather than microcontroller deployment.
Abstract
Large speech models are rapidly gaining traction in research community. As a result, model compression has become an important topic, so that these models can fit in memory and be served with reduced cost. Practical approaches for compressing automatic speech recognition (ASR) model use int8 or int4 weight quantization. In this study, we propose to develop 2-bit ASR models. We explore the impact of symmetric and asymmetric quantization combined with sub-channel quantization and clipping on both LibriSpeech dataset and large-scale training data. We obtain a lossless 2-bit Conformer model with 32% model size reduction when compared to state of the art 4-bit Conformer model for LibriSpeech. Wit…
Prior work it builds on
-
Zipformer: A faster and better encoder for automatic speech recognitionZengwei Yao, Liyong Guo, Xiao-Yu Yang et al. · 2023 · International Conference on Learning Representations · cited 184 times · Not summarized yet
Zipformer introduces the encoder architecture used by Kroko, the recognizer quantized and deployed in this paper.
Abstract
The Conformer has become the most popular encoder model for automatic speech recognition (ASR). It adds convolution modules to a transformer to learn both local and global dependencies. In this work we describe a faster, more memory-efficient, and better-performing transformer, called Zipformer. Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows us to retain some length information; 4) new activation functions SwooshR and SwooshL work better than Swish. We also propose a new…
-
Moonshine: Speech Recognition for Live Transcription and Voice CommandsNat Jeffries, Evan King, Manjunath Kudlur et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 31 times · Not summarized yet
Moonshine develops efficient speech-recognition models for live transcription and voice commands, but does not address low-bit microcontroller deployment.
Abstract
This paper introduces Moonshine, a family of speech recognition models optimized for live transcription and voice command processing. Moonshine is based on an encoder-decoder transformer architecture and employs Rotary Position Embedding (RoPE) instead of traditional absolute position embeddings. The model is trained on speech segments of various lengths, but without using zero-padding, leading to greater efficiency for the encoder during inference time. When benchmarked against OpenAI's Whisper tiny-en, Moonshine Tiny demonstrates a 5x reduction in compute requirements for transcribing a 10-second speech segment while incurring no increase in word error rates across standard evaluation data…
-
Librispeech: An ASR corpus based on public domain audio booksVassil Panayotov, Guo-Guo Chen, D. Povey et al. · 2015 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cited 8,710 times · Not summarized yet
Librispeech provides the audiobook speech corpus used to evaluate the recognizer’s word error rate.
-
Robust Speech Recognition via Large-Scale Weak SupervisionA. Radford, Jong Wook Kim, Tao Xu et al. · 2022 · International Conference on Machine Learning · cited 9,370 times · Not summarized yet
Robust Speech Recognition studies large-scale weakly supervised models and their benchmark accuracy, rather than low-bit embedded inference.
Abstract
We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
-
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersElias Frantar, Saleh Ashkboos, T. Hoefler et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 2,863 times · AI summary available
GPTQ presents the post-training weight-quantization method used in this paper to round the encoder’s quantized weights.
Abstract
Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization met…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 2 November 2026.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.