JuicyTalk
Tencent has released AngelSpec, an open-source, torch-native training framework for speculative-decoding draft models. The release covers both autoregressive multi-token prediction (MTP) and the block-parallel DFlash family.
Most speculative-decoding work searches for one drafter that scores well on an averaged benchmark mixture. Real serving traffic does not look like that mixture. AngelSpec treats workload heterogeneity as a first-class design constraint, and specializes structure, training data, and verification depth around it.
Why one universal drafter underperforms
Speculative decoding is lossless. A lightweight drafter proposes several future tokens, and the target model verifies them together in one forward pass using rejection sampling. Acceleration then depends on two things: how many draft tokens get accepted, and how long the complete draft–verify round takes.
Those two quantities move in opposite directions across domains. In high-entropy open-ended conversation, many continuations are semantically valid. The target may pick any one of them, so acceptance decays quickly with proposal depth. Generating and verifying a long block wastes compute. Autoregressive MTP drafting fits this regime because it proposes a shorter candidate sequence.
Code and mathematical reasoning behave differently. Programming syntax, repeated identifiers, formal expressions, and step-by-step derivations constrain future tokens more strongly. These workloads create longer predictable spans, which is exactly what block-parallel drafting amortizes well.
AngelSpec therefore ships two complementary drafters, not one compromise. The MTP model is trained on rich, diverse conversation-oriented data. The block-diffusion model is strengthened with code- and mathematics-focused samples.
The MTP path: Training-Time Test and target-model rollout
The original Hy3 model is trained with a single MTP layer and no recurrent self-conditioned unrolling. At inference the block can be reused recurrently, but the training objective never prepared it for a long self-generated chain. Errors accumulate with depth, so the second and third draft positions accept at substantially lower rates than the first.
AngelSpec addresses this train–inference mismatch with a shared-parameter, multi-depth scheme. It retains D logical prediction depths but reuses one physical MTP block. During training that block is autoregressively unrolled for D steps, with each prediction fed into the next invocation. Following the Training-Time Test principle of EAGLE-3, depth k+1 receives the arg max prediction from depth k instead of the ground-truth token. Parameters are shared, but supervision stays depth-specific: the teacher target advances one future position at every depth.
Two further choices carry most of the gain. First, the target backbone and target language-model head are frozen, and MTP inputs from the backbone are detached. The drafter improves its proposal distribution without touching the distribution that verification must preserve. Second, training uses target-model rollout: responses are generated by the frozen target rather than taken from the original reference corpus. That produces the exact token choices, hidden-state trajectories, and local uncertainty patterns MTP has to approximate at serving time.
The measured effect is concentrated where it should be. At T = 0, mean acceptance moves from 52.8% to 66.4%, and mean accepted length from 2.58 to 2.99. The first-position rate is nearly preserved at 0.799 → 0.814. The deeper positions carry the delta: p3 climbs from 0.290 to 0.706 on GSM8K, and from 0.387 to 0.757 on HumanEval.
” + bars([
[“MTP”,3.24],[“DFlash”,4.57],[“DSpark”,5.32],[“DFly”,5.41]
],6.5,”DFly”);
h += ‘
Target: Hy3-A21B
‘ + bars([
[“MTP”,3.00],[“DFlash”,3.69],[“DFly”,4.79]
],6.5,”DFly”);
h += ‘
+59.7%DFly vs MTP on Hy3-A21B
‘
+ ‘
+29.8%DFly vs DFlash on Hy3-A21B
‘
+ ‘
6.42DFly MAL on GSM8K (Qwen3-8B)
‘;
h += ‘
Per-benchmark MAL
| Drafter | Math500 | GSM8K | HumanEval | MBPP | LiveCodeBench | MT-Bench | Avg |
|---|
‘;
h += ‘
DSpark keeps the lead on MT-Bench for Qwen3-8B (3.77 vs 3.67). That is consistent with the report positioning DFly for code and math workloads.
‘;
return h;
}
},
tput:{
cmd:”angelspec bench –throughput –model hy3-295b-a21b –tp 8 –temp 1″,
note:”Output-token throughput on HY3-295B-A21B, TP=8, averaged over six datasets. Each cell uses 3 × 120 s windows. Pick a concurrency level.”,
src:’Source: Table 7, AngelSpec technical report.’,
render:function(){
return ‘
‘;
},
after:function(){
var box = document.getElementById(“asxConc”);
box.addEventListener(“click”, function(e){
var b = e.target.closest(“button”); if(!b) return;
Array.prototype.forEach.call(box.querySelectorAll(“button”),function(x){x.classList.remove(“on”);});
b.classList.add(“on”); drawTput(b.getAttribute(“data-c”));
});
drawTput(“c16″);
}
},
ablate:{
cmd:”angelspec ablate –arch dfly –target hy3-a21b –greedy”,
note:”Cumulative ablation on Hy3-A21B under greedy decoding (T = 0, no thinking). All variants use D-PACE. Each row adds one change to the row above.”,
src:’Source: Table 4, AngelSpec technical report.’,
render:function(){
var h=”
Average mean accepted length
” + bars([
[“DFlash backbone”,3.77],[“+ DFly backbone”,4.40],[“+ Markov head”,4.56],
[“+ Hidden correction”,4.60],[“+ Code/Math data”,4.75]
],5.2,”+ Code/Math data”);
h += ‘
Where the gain lands
| Step | Math500 | GSM8K | HumanEval | MBPP | LiveCodeBench | MT-Bench | Avg |
|---|
‘;
h += ‘
The domain data expansion adds 700K prompts: 500K code from OpenCodeInstruct and OpenCodeReasoning, 200K math from Big-Math. MT-Bench slips from 3.06 to 3.02, which the report calls the intended specialization trade.
‘;
return h;
}
},
mtp:{
cmd:”angelspec mtp –ttt –target-rollout –depth 3″,
note:”Hy3 MTP acceptance before and after Training-Time Test on the target-model rollout mixture. Avg is the mean of the three cumulative position-wise acceptance rates.”,
src:’Source: Table 2 and Section 2.5, AngelSpec technical report.’,
render:function(){
var h=”
+ ‘
52.8% → 66.4%Mean acceptance, T = 0
‘
+ ‘
2.58 → 2.99Mean accepted length, T = 0
‘
+ ‘
51.3% → 63.3%Mean acceptance, T = 0.9
‘
+ ‘
2.54 → 2.90Mean accepted length, T = 0.9
‘;
h += ‘
Per-position acceptance, T = 0
‘ + pair([
[“p1”,0.799,0.814],[“p2”,0.518,0.653],[“p3”,0.266,0.524]
]);
h += ‘
Per-position acceptance, T = 0.9
‘ + pair([
[“p1”,0.789,0.792],[“p2”,0.499,0.619],[“p3”,0.253,0.486]
]);
h += ‘
The first proposal was already strong and is nearly preserved. The gain is concentrated in the speculative suffix: p3 moves from 0.290 to 0.706 on GSM8K, and from 0.387 to 0.757 on HumanEval.
‘;
h += ‘
Loss ablation (small-scale MoE target, D = 4)
‘ + bars([
[“CE Loss”,3.80],[“KL Loss”,3.92],[“LK Loss”,3.95],[“TV Loss”,3.03],[“e2e TV + LK cold start”,3.96]
],4.2,”e2e TV + LK cold start”);
return h;
}
},
dcut:{
cmd:”angelspec dcut –replay hunyuan-live –gpus 8xH20 –conc 2-64″,
note:”D-cut replayed against Hunyuan production traffic on Hy3-295B-A21B, TP=8, 8× H20. Aggregate throughput in tok/s at the high-load end of the sweep.”,
src:’Source: Section 4.4 and Figure 4, AngelSpec technical report.’,
render:function(){
var h=”
Aggregate throughput (tok/s)
| Concurrency | DFly-8 | DFly-8 + D-cut | Delta |
|---|---|---|---|
| c48 | 860 | 886 | +3.0% |
| c56 | 858 | 937 | +9.2% |
| c64 | 848 | 981 | +15.7% |
‘;
h += ‘
Speedup over autoregressive decoding
‘ + bars([
[“DFly peak (c24-c40)”,1.33],[“DFly at c64”,1.25],[“D-cut at c56”,1.45],[“D-cut at c64″,1.44]
],1.6,”D-cut at c56″,”×”);
h += ‘
+ ‘
2.50 → 2.46Mean accepted length over the sweep
‘
+ ‘
−1.5%Acceptance given up on average
‘
+ ‘
14%Extra throughput at matched 15.3 tok/s per user
‘;
h += ‘
D-cut restricts the verification budget to the ratios 0.25, 0.50, 0.75 and 1.00, then picks the ratio that maximises estimated batch progress divided by profiled step latency. At c64 acceptance falls only from 2.50 to 2.43 while throughput rises 15.7%.
‘;
return h;
}
},
prof:{
cmd:”angelspec profile –nsight –dataset math-500 –seed 1234″,
note:”Break-even committed length: the minimum committed tokens per iteration a method needs to match the predicted TPS of Hy3-MTP3 at the same request concurrency. Lower is easier to beat.”,
src:’Source: Table 8 and Section 5, AngelSpec technical report.’,
render:function(){
var h=”
Average break-even committed length (tokens)
” + bars([
[“Hy3-MTP3 (baseline)”,3.00],[“DFlash L3-B5”,3.80],[“DFlash L5-B8”,4.61],[“DFlash L7-B8”,4.56],
[“DFly backbone”,3.67],[“DFly + Markov”,5.21],[“DFly (final)”,5.14]
],5.6,”Hy3-MTP3 (baseline)”);
h += ‘
+ ‘
19.77-64.16 msMedian execute_model duration
‘
+ ‘
0.89-4.49 msMedian sample_tokens duration
‘
+ ‘
50.54 msDFly avg total latency proxy
‘
+ ‘
51.46 msBackbone + Markov latency proxy
‘;
h += ‘
Target-model execution dominates device-side latency, so proposer depth is not the main cost driver. Hidden correction lands at a lower average latency proxy than the Markov head, which is why it ships as the default in DFly.
‘;
return h;
}
}
};
var TP = {
c4:[[“AR”,288.4,1.00],[“MTP-3”,460.2,1.60],[“DFlash-8”,516.9,1.79],[“DFly-8”,571.2,1.98]],
c8:[[“AR”,421.8,1.00],[“MTP-3”,701.0,1.66],[“DFlash-8”,767.4,1.82],[“DFly-8”,851.2,2.02]],
c16:[[“AR”,638.2,1.00],[“MTP-3”,1045.0,1.64],[“DFlash-8”,1270.3,1.99],[“DFly-8”,1418.8,2.22]],
c32:[[“AR”,933.9,1.00],[“MTP-3”,1736.9,1.86],[“DFlash-8”,2008.2,2.15],[“DFly-8”,2243.1,2.40]],
c64:[[“AR”,1297.7,1.00],[“MTP-3”,2692.2,2.08],[“DFlash-8”,2438.2,1.89],[“DFly-8”,2726.2,2.11]]
};
function esc(s){return String(s);}
function bars(rows,max,winner,suffix){
suffix = suffix || “”;
return rows.map(function(r){
var pct = Math.max(3, Math.min(100, (r[1]/max)*100));
var win = (r[0]===winner);
return ‘
‘+esc(r[0])+’
‘
+ ”
+ ‘
‘+r[1].toFixed(2)+suffix+’
‘;
}).join(“”);
}
function pair(rows){
return rows.map(function(r){
var b = Math.max(3,r[1]*100), a = Math.max(3,r[2]*100);
return ‘
‘+r[0]+’ base
‘
+ ”
+ ‘
‘+r[1].toFixed(3)+’
‘
+ ‘
‘+r[0]+’ TTT
‘
+ ”
+ ‘
‘+r[2].toFixed(3)+’
‘;
}).join(“”);
}
function row(label,vals,win){
return ‘
‘
+ vals.map(function(v){return ‘
‘;}).join(“”) + ‘
‘;
}
function drawTput(c){
var d = TP[c], max = 2900;
var h=”
Average output-token throughput at “+c+’
‘;
h += d.map(function(r){
var pct = Math.max(3,(r[1]/max)*100), win = (r[0]===”DFly-8″);
return ‘
‘+r[0]+’
‘
+ ”
+ ‘
‘+r[1].toFixed(1)+’
‘;
}).join(“”);
h += ‘
+ ‘
‘+d[3][2].toFixed(2)+’×DFly-8 speedup over AR at ‘+c+’
‘
+ ‘
‘+d[2][2].toFixed(2)+’×DFlash-8 speedup over AR at ‘+c+’
‘
+ ‘
‘+d[1][2].toFixed(2)+’×MTP-3 speedup over AR at ‘+c+’
‘
+ ‘
+’+(((d[3][1]/d[2][1])-1)*100).toFixed(1)+’%DFly-8 over DFlash-8 at ‘+c+’
‘;
if(c===”c64″){
h += ‘
At c64 the picture is mixed by dataset: MTP-3 reaches 2.53× on GSM8K against 2.44× for DFly-8, and leads MT-Bench at 1.80× against 1.48×. DFly-8 still holds the highest six-dataset average.
‘;
} else {
h += ‘
DFly-8 posts the highest average speedup at every tested concurrency from 4 to 64, and 10.5% to 11.8% higher average throughput than DFlash-8 across the sweep.
‘;
}
document.getElementById(“asxTputBody”).innerHTML = h;
}
var promptEl = document.getElementById(“asxPrompt”);
var outEl = document.getElementById(“asxOut”);
var srcEl = document.getElementById(“asxSrc”);
function ping(){
try{ parent.postMessage({type:”mtp-resize-asx1″, height: document.body.offsetHeight + 40}, “*”); }catch(e){}
}
function run(k){
var d = D[k];
promptEl.innerHTML = ‘$ ‘+d.cmd+’ ‘;
outEl.innerHTML = ‘
‘+d.note+’
‘ + d.render();
srcEl.innerHTML = d.src;
if(d.after) d.after();
setTimeout(ping,40); setTimeout(ping,700);
}
document.addEventListener(“click”, function(e){
var b = e.target.closest(“.asx-cmd”); if(!b) return;
Array.prototype.forEach.call(document.querySelectorAll(“.asx-cmd”),function(x){x.classList.remove(“on”);});
b.classList.add(“on”); run(b.getAttribute(“data-k”));
});
run(“quality”);
window.addEventListener(“load”, ping);
window.addEventListener(“resize”, ping);
setInterval(ping, 1600);
})();
