Speculative decoding: What holds it back and how to make it faster

What is it about?

Speculative decoding offers large language models (LLMs) a shortcut to faster responses: draft ahead, then check several predictions together. Yet drafting costs, rejected text and waiting between stages can eat into the savings. Our review examines these limitations and maps them to corresponding optimization methods across drafting, verification and system coordination. It explains why improving one step does not always speed up the whole process, and highlights paths toward further progress.

Featured Image

Photo by Google DeepMind on Unsplash

Photo by Google DeepMind on Unsplash

Why is it important?

The speed that matters is the speed users actually experience. Our review connects the limitations of speculative decoding with corresponding optimization methods, showing why gains in one part can be lost elsewhere. This gives researchers and developers a clearer basis for deciding what to improve, which trade-offs to consider and where further research is needed. That perspective is particularly valuable for longer outputs and many simultaneous requests, helping guide progress toward LLM inference speedups that hold up in practice.