Bubble-gum thinking: ideas worth chewing on, not necessarily worth swallowing. This one — generalize token-level speculative decoding (MTP, DFlash) from individual tokens to whole utterance chunks. Same shape as CPU branch prediction, just operating at the dialog layer instead of the instruction stream. 6.2× faster mouth-to-mouth, 16× less jitter on a commodity stack. Probably not novel; definitely measurable.
Same question, same models, same hardware. First half = speculative (~56ms gap). Then 1.5s pause. Second half = baseline (~240ms gap). Listen for the silence after "today?":
The 184ms difference is right at the conscious-perception threshold for "is the system thinking?" Baseline reads as "the system is processing." Speculative reads as "responsive."
Voice agent pipelines are serial: VAD → ASR → LLM → TTS. Each stage gates the next.
Total mouth-to-mouth latency = sum of stages, and the LLM first-token step dominates (~200ms on a
small model, more on bigger ones).
Token-level speculative decoding (Medusa, DFlash, MTP) works because a draft model proposes tokens in parallel that a target model verifies in one forward pass. The amortization beats serial autoregressive generation.
At partial-ASR time (before the user finishes speaking), fork the LLM into K parallel branches on different hypothesis completions. When ASR finalizes, pick the closest match via cosine similarity in embedding space and commit. The winning branch's tokens are already streaming — downstream TTS starts immediately. Losers get cancelled.
| Mode | Mean | Median | Stdev | Range |
|---|---|---|---|---|
| Speculative | 54.18ms | 51.65ms | 6.2ms | 49–66ms |
| Baseline | 337.21ms | 293.66ms | 102ms | 239–559ms |
n=8 paired trials (4 utterances × 2 reps). End-to-end stacked benchmark, real chunked-input pacing, real streaming partials.
Voice UX is dominated by latency variance, not mean. A predictable 54ms beats a jittery 337ms even if both were "good enough." Baseline ranges 239–559ms because ASR, LLM first-token, and TTS render each have independent jitter that compounds. Speculative absorbs all three by running them in parallel during the speech window.
small.enqwen3.5:0.8ben_US-lessac-mediumOne consumer GPU (RTX 5090 Laptop) for whisper + embedder, one server GPU (RTX PRO 6000) for LLM. Nothing exotic. The interesting work is the orchestration, not the components.
The cosine-sim picker doesn't always select the "right" hypothesis. In one demo run, partial-ASR
transcribed "weather" as "way" mid-stream, so the picked hypothesis was the gibberish
"What's the way? like today". But the LLM running on it produced:
A perfectly fine answer to the actual question. Speculation tolerates bad hypotheses because adjacent-intent LLM responses still answer the user's actual intent. The picker selects the closest available; "closest-but-wrong" is usually still useful.
This is the kind of thing that makes a brittle-looking system surprisingly robust. The brittleness of the predictor stops mattering once the LLM is downstream.
If you've published or built this combination, please open an issue with a pointer. Filed as bubble-gum thinking precisely because I don't know whether it's novel.
OLLAMA_NUM_PARALLEL. Even so, average advance was 8–27 tokens per branch by final-ASR time.Earlier sub-experiment: pre-render a "graceful pivot" utterance alongside the primary response. On interrupt, swap to the pre-rendered pivot PCM instantly (vs. 42ms baseline to render-on-interrupt).
Hard-cut click at the swap is intentional and audible — prosody-aware crossfade is Phase 4 work, not done here. The point is the structural latency saving, not the audio polish.
DFlash and MTP have always felt like they were operating at the wrong abstraction level. Tokens are too granular for a lot of what makes voice agents feel slow. The slowness lives at boundaries: ASR-final → LLM-start, LLM-done → TTS-start. Each boundary is a serial gate where speculation could amortize the waiting.
Whether or not anyone ships this exact stack, the framing — "fork at uncertainty, commit at resolution, in parallel during the speech window" — feels like a useful mental model for designing voice systems. Even if it turns out to be reinvention of prior art, it's reinvention I needed to do to understand the shape.
Open-source on GitHub. Reproduces in one session on any reasonable GPU setup. License: MIT.