DeepSpec: A Production Guide to Speculative Decoding

Inference optimization often gets reduced to one number: tokens per second. DeepSpec, a new full-stack codebase from DeepSeek, is a reminder that the number is produced by a system. Its repository covers data preparation, draft-model training, released checkpoints, and evaluation for speculative decoding rather than presenting a single kernel or benchmark chart.

Speculative decoding pairs a smaller draft model with a larger target model. The draft proposes multiple tokens; the target verifies them in fewer expensive passes. When enough proposed tokens are accepted, users see lower latency without changing the target model’s output distribution. When acceptance is poor, the extra draft work can erase the benefit.

Source reviewed for this guide: DeepSpec official repository and its linked DSpark paper.

DeepSpec’s three-stage workflow

The official workflow is intentionally sequential:

prepare data and regenerate target answers;
train a draft model against a target cache;
evaluate acceptance on representative tasks.

That order is not administrative. Each stage defines the next stage’s validity. A cache produced with the wrong target settings trains a draft model for a system you do not deploy. A benchmark unrelated to production traffic can make an impressive checkpoint look useful when its acceptance rate collapses on real prompts.

DeepSpec currently includes DSpark, DFlash, and Eagle3 implementations. It also publishes checkpoints for Qwen3 4B, 8B, and 14B targets and Gemma 4 12B IT. Those checkpoints are valuable baselines, but the repository cautions that domain-specific deployments should fine-tune again—especially when the target operates in thinking mode.

Start with the economics, not the training command

The default data-preparation path can require roughly 38 TB of target cache for the documented Qwen3-4B setting. The default training scripts assume one node with eight visible GPUs. These are not incidental details. Before cloning the repository, estimate four budgets:

storage for prompts, regenerated answers, cache shards, and checkpoints;
target-model inference required to build the cache;
GPU hours for draft training;
engineering time for integration and repeatable evaluation.

If your serving bill is small or traffic is highly variable, buying this pipeline may never pay back. If you serve a stable, high-volume workload where each millisecond matters, a domain-tuned draft model can have durable value.

A simple decision model is:

monthly benefit = requests × tokens/request × latency value × measured speedup
monthly cost = amortized training + storage + extra draft serving + maintenance

Do not substitute a paper speedup for measured speedup. It depends on acceptance length, hardware, batch shape, target mode, and the prompt distribution.

Explore DeepSeek in MidassAI

Build a representative evaluation slice

DeepSpec includes GSM8K, Math500, AIME25, HumanEval, MBPP, LiveCodeBench, MT-Bench, Alpaca, and Arena-Hard v2. This range helps compare algorithms, but production readiness needs your own slice.

Sample traffic by task, output length, language, context size, and tool-use pattern. Remove sensitive data and preserve structural characteristics. A coding service might divide the set into completion, refactor, test generation, explanation, and repository-level reasoning. A support assistant might divide it by intent and conversation depth.

For each slice, record:

accepted tokens per verification step;
end-to-end time to first token and total completion time;
draft and target GPU utilization;
output equivalence checks;
fallback behavior when draft inference fails;
peak memory and cache pressure.

An average can hide a damaging tail. If short chat answers speed up while long code generation slows down, route speculative decoding only to the segment where it wins.

Match target behavior exactly

The draft learns a proposal distribution for a particular target. Regenerate training answers using the same target checkpoint, tokenizer, decoding configuration, and thinking mode used in production. Seemingly small drift changes acceptance.

Version the full pairing, not just the draft checkpoint:

target model + target revision + tokenizer + sampling policy
+ draft model + draft revision + DeepSpec config + training data snapshot

When the target changes, rerun a compatibility evaluation before reusing the draft. A draft trained on non-thinking outputs should not be assumed to accelerate thinking-mode traffic. DeepSpec’s own guidance calls this distinction out.

Use the released checkpoints as controls

Released DSpark, DFlash, and Eagle3 checkpoints provide a useful experiment ladder. First reproduce evaluation with a published pairing. Next run the same checkpoint against your sanitized traffic. Only then train a custom draft.

This separates environment problems from data problems. If the official pairing does not reproduce, inspect software versions, GPU architecture, tokenizer alignment, and evaluation settings. If it reproduces but performs poorly on your traffic, custom training may help. If a custom draft still has weak acceptance, your workload may simply be a poor fit.

Deploy with a reversible router

Do not place speculative decoding in the only path to the target model. Put it behind a traffic router with a direct-target fallback. Start with shadow evaluation, then a small percentage of live traffic. Log acceptance metrics without retaining sensitive prompts.

Rollback should require a configuration change, not a rebuild. Watch both speed and quality signals because operational defects—tokenizer mismatch, stale caches, memory pressure—can appear as latency spikes or malformed output.

DeepSpec makes speculative decoding more approachable by publishing the whole lifecycle. It also makes the true cost visible. The practical opportunity is not “free speed.” It is a controlled exchange: invest in data, training, and evaluation to reduce repeated target-model work on a predictable traffic distribution. Teams that measure that exchange carefully can turn a research technique into a useful serving layer; teams that skip the measurement will only add another model to operate.