🫧 Bubble-gum thinking

SPECVOICE: speculative execution at the dialog layer

2026-05-13 · Weekend prototype · github.com/scrappylabsai/specvoice

Bubble-gum thinking: ideas worth chewing on, not necessarily worth swallowing. This one — generalize token-level speculative decoding (MTP, DFlash) from individual tokens to whole utterance chunks. Same shape as CPU branch prediction, just operating at the dialog layer instead of the instruction stream. 6.2× faster mouth-to-mouth, 16× less jitter on a commodity stack. Probably not novel; definitely measurable.

01The idea

Voice agent pipelines are serial: VAD → ASR → LLM → TTS. Each stage gates the next. Total mouth-to-mouth latency = sum of stages, and the LLM first-token step dominates (~200ms on a small model, more on bigger ones).

Token-level speculative decoding (Medusa, DFlash, MTP) works because a draft model proposes tokens in parallel that a target model verifies in one forward pass. The amortization beats serial autoregressive generation.

What if you do that at the dialog layer? The "draft tokens" become whole candidate utterances.

At partial-ASR time (before the user finishes speaking), fork the LLM into K parallel branches on different hypothesis completions. When ASR finalizes, pick the closest match via cosine similarity in embedding space and commit. The winning branch's tokens are already streaming — downstream TTS starts immediately. Losers get cancelled.

02The pipeline

streaming ASR partials ─┬─► LLM branch 1 (h1: "What's the weather")        ─► Piper TTS render 1
                          ├─► LLM branch 2 (h2: "...right now")                 ─► Piper TTS render 2
                          └─► LLM branch 3 (h3: "...like today")                ─► Piper TTS render 3
                                       │
                                       ▼
                          on final-ASR:  embed final → cosine-sim → pick closest hypothesis
                                       │
                                       ▼
                          audio out  (winner's PCM already in memory)

03What the numbers say

n=8 paired trials (4 utterances × 2 reps). End-to-end stacked benchmark, real chunked-input pacing, real streaming partials.

The jitter story is more interesting than the speedup

Voice UX is dominated by latency variance, not mean. A predictable 54ms beats a jittery 337ms even if both were "good enough." Baseline ranges 239–559ms because ASR, LLM first-token, and TTS render each have independent jitter that compounds. Speculative absorbs all three by running them in parallel during the speech window.

04The stack

One consumer GPU (RTX 5090 Laptop) for whisper + embedder, one server GPU (RTX PRO 6000) for LLM. Nothing exotic. The interesting work is the orchestration, not the components.

05The unexpected finding

The cosine-sim picker doesn't always select the "right" hypothesis. In one demo run, partial-ASR transcribed "weather" as "way" mid-stream, so the picked hypothesis was the gibberish "What's the way? like today". But the LLM running on it produced:

"It is a lovely day!"

A perfectly fine answer to the actual question. Speculation tolerates bad hypotheses because adjacent-intent LLM responses still answer the user's actual intent. The picker selects the closest available; "closest-but-wrong" is usually still useful.

This is the kind of thing that makes a brittle-looking system surprisingly robust. The brittleness of the predictor stops mattering once the LLM is downstream.

06What's new, what's not

Probably not new

• Streaming-ASR-with-partial-LLM-dispatching — commercial voice assistants have been doing variants since ~2018
• Speculative decoding (token-level) — well-studied: Medusa, DFlash, MTP
• Cosine-sim ranking — basic embedding-space comparison
• Pre-fetching candidate responses — Google "barge-in," Alexa likely similar internals

What I haven't seen written up cleanly

• Fork-and-commit stacked across LLM + TTS in a single open prototype
• Cosine-sim commit picker on a small hypothesis bank, measured end-to-end
• An audible A/B artifact showing what the latency difference sounds like
• The "imperfect-hypothesis tolerance" observation, even though it's probably folklore

If you've published or built this combination, please open an issue with a pointer. Filed as bubble-gum thinking precisely because I don't know whether it's novel.

07Honest caveats

• No real intent predictor. Hypotheses are heuristic suffixes. Production needs a small classifier or n-gram LM completion to generate diverse plausible candidates.
• No live VAD endpoint detection. "Speech ends" is a fixed silence pad. Production needs Silero-VAD-driven endpoints.
• No prosody-aware stitching. Phase 1 hard-cuts at swap. Audible click (hear it in the Phase 1 artifact below). Phase 4 territory.
• K=3 compute cost. Three parallel LLM gens per utterance. Self-hosted: "free." API: 3×.
• n=8 trials. Small sample. Numbers reproduce but absolute mean shifts with hardware/network.
• Ollama partially serializes K=3 concurrent gens without explicit OLLAMA_NUM_PARALLEL. Even so, average advance was 8–27 tokens per branch by final-ASR time.

08Phase 1 bonus: TTS pivot speculation

09Why I built this

DFlash and MTP have always felt like they were operating at the wrong abstraction level. Tokens are too granular for a lot of what makes voice agents feel slow. The slowness lives at boundaries: ASR-final → LLM-start, LLM-done → TTS-start. Each boundary is a serial gate where speculation could amortize the waiting.

Whether or not anyone ships this exact stack, the framing — "fork at uncertainty, commit at resolution, in parallel during the speech window" — feels like a useful mental model for designing voice systems. Even if it turns out to be reinvention of prior art, it's reinvention I needed to do to understand the shape.

10What's filed

Open-source on GitHub. Reproduces in one session on any reasonable GPU setup. License: MIT.

→ Code on GitHub → Download A/B WAV