Why is Speculative Decoding Fast?

Published 12 October 2026

A small draft model proposes tokens for a larger model to verify in parallel, helping use otherwise idle compute to generate sequences faster at low batch sizes. At high throughput, the extra computation may be less efficient than moving memory.