Mesh decode performance budget — what's required to sustain 15 tok/s (+ 20, 30)
mistral-7b-instruct-v0.3 on a Cloud Run L4 (24GB, ~300 GB/s HBM peak, 8 vCPU). Single-token decode. All numbers measured this session (2026-06-21) unless marked "floor" (theoretical) or "target".
The core fact: decode is bandwidth-bound on reading the weights once per
token. So tok/s is governed by bytes read per token ÷ achieved bandwidth.
Everything below follows from that.
0. WHERE THE TIME GOES (per token) — the one table
Combines components × bytes (§2) × per-lane achieved bandwidth (constraints doc).
ms are measured (CPU=00059/00068, cuBLAS-f16=00065, Q8-naive=00070); Q8-opt is
projected at a conservative ~200 GB/s achieved (kernel building; ceiling ~250).
| component | Q8 bytes | f16 bytes | CPU ms | GPU f16 ms | GPU Q8-naive ms | GPU Q8-opt ms |
|---|---|---|---|---|---|---|
| QKV proj | 0.86 GB | 1.61 GB | 18.0 | 16.8 | 17.0 | 4.3 |
| O proj | 0.57 GB | 1.07 GB | 9.0 | 8.6 | 9.0 | 2.9 |
| FFN gate+up+down | 5.99 GB | 11.3 GB | 60.0 | 59.9 | 70.0 | 30.0 |
| logit (lm_head) | 0.14 GB | 0.27 GB | 1.3 | 1.3 | 1.5 | 0.7 |
| matvec subtotal | 7.55 GB | 14.2 GB | 88 | 87 | 97 | 38 |
| attention (KV, CPU) | — | — | 3.6 | 3.6 | 3.6 | 3.6 |
| norms/rope/swiglu | — | — | 4.0 | 4.0 | 4.0 | 4.0 |
| response overhead | — | — | 3.0 | 3.0 | 3.0 | 3.0 |
| forward / token | ~99 ms | ~98 ms | ~108 ms | ~49 ms | ||
| achieved BW | 82 GB/s | 164 GB/s | 86 GB/s | ~200 GB/s | ||
| tok/s | 10.2 | 10.1 | 9.0 | ~20 (proj) |
Reading: FFN is ~80% of the matvec (5.99 of 7.55 GB) — optimize FFN-shaped GEMVs first. cuBLAS-f16 ties CPU (reads 2× bytes at 2× BW). Q8-naive loses (fewer bytes, but only 86 GB/s). Q8-opt wins by reading half the bytes at high %-of-peak. The 15-tps line is matvec ≤56ms + ~11ms non-matvec = 67ms (see §1).
1. The time budget (forward ms = 1000 / tok·s)
| target | forward/token | matvec budget* |
|---|---|---|
| 10 tps | 100.0 ms | ~89 ms |
| 15 tps | 66.7 ms | ~56 ms |
| 20 tps | 50.0 ms | ~39 ms |
| 30 tps | 33.3 ms | ~22 ms |
*Non-matvec floor ≈ ~11 ms/token: attention ~3.6ms (post Wave-10), RMSNorm/rope/ swiglu glue ~3ms, logit ~1.3ms, per-call/response overhead ~3ms. matvec budget = forward − 11ms. This is the wall the weight-read must fit under.
2. Weight bytes read per token (the denominator that matters)
Per layer (out×in params): q 16.78M, k 4.19M, v 4.19M, o 16.78M, gate 58.72M, up 58.72M, down 58.72M = 218.1M params/layer × 32 layers + lm_head 134M ≈ 7.11B params read per token. Bytes by quant:
| dtype | bytes/param | total/token | 300 GB/s floor |
|---|---|---|---|
| f16 | 2.000 | 14.2 GB | 47 ms |
| Q8_0 | 1.0625 | 7.55 GB | 25 ms |
| Q6_K | ~0.82 | ~5.8 GB | 19 ms |
| Q5_K_M | ~0.69 | ~4.9 GB | 16 ms |
| Q4_K_M | ~0.5625 | ~4.0 GB | 13 ms |
3. THE REQUIREMENT MATRIX — effective bandwidth needed (GB/s) to read the
weights inside the matvec budget, per dtype × target. Compare to what each execution lane actually achieves (§4). Bold = the binding question for 15 tps.
| dtype | 15 tps (≤56ms) | 20 tps (≤39ms) | 30 tps (≤22ms) |
|---|---|---|---|
| f16 | 254 GB/s | 364 (>peak ✗) | impossible |
| Q8_0 | 135 GB/s | 194 GB/s | 343 (>peak ✗) |
| Q6_K | 104 GB/s | 149 GB/s | 264 GB/s |
| Q4_K_M | 71 GB/s | 103 GB/s | 182 GB/s |
4. Achieved bandwidth per execution lane (MEASURED) — vs the requirement
| lane | reads | achieved BW | matvec/token | tps | notes |
|---|---|---|---|---|---|
| CPU (Q8 scalar+AVX2, 8 vCPU) | 7.55 GB | ~82 GB/s | ~92 ms | 10.2 | prod baseline 00059/00068. NOT bandwidth-bound in the HBM sense — latency/compute-bound; Q4 on CPU == Q8 (K-quant dequant offsets fewer bytes). 8-vCPU hard cap. |
| GPU cuBLAS f16 (slab, 00065) | 14.2 GB | ~183 GB/s | ~78 ms | 10.1 | full residency works; reads 2× the bytes → can't clear 15 (needs 254 GB/s on f16, >cuBLAS). |
| GPU Q8 kernel — naive (00070) | 7.55 GB | ~86 GB/s | ~70-88 ms | 9.0 | correct (divergent=0) but memory-inefficient (half-idle threads on K=4096, byte loads) → fewer bytes wasted. |
| GPU Q8 kernel — optimized (target) | 7.55 GB | ≥250 GB/s target | ~30 ms | ~22-25 | warp-per-row + float4 loads (building now). Clears 15 with margin IF it hits ~135+ GB/s (very achievable — that's <half the naive-kernel's theoretical headroom). |
| GPU Q4_K kernel (future) | ~4.0 GB | ≥71 GB/s for 15 | ~16 ms @250 | ~30+ | lowest bar to 15 (71 GB/s — even the naive 86 GB/s kernel clears it). Quality tradeoff (4-bit). q4km knot already in R2. |
5. Verdict — the path per target
- 10 tps: DONE (CPU baseline, full Q8 quality, prod
00068-f45). - 15 tps:
- f16 cuBLAS: NO (needs 254 GB/s, hardware gives ~183).
- Q8 optimized kernel: YES (needs only 135 GB/s; the optimized kernel targets ~250). ← in flight, no quality loss.
- Q4_K kernel: easiest (needs 71 GB/s) but 4-bit quality cost.
- 20 tps: Q8 optimized (needs 194 GB/s — achievable on the L4 with a good kernel) OR Q4_K. f16 is OUT (364 > peak).
- 30 tps: ONLY Q4_K (Q8 needs 343 GB/s ≈ HBM peak, no headroom; Q4 needs 182).
6. What each component contributes (per-GEMV, so you know where the ms go)
Measured warm, GPU lane (cuBLAS f16 / Q8-naive ms):
| proj | shape | f16 bytes | cuBLAS f16 | Q8 naive |
|---|---|---|---|---|
| gate | [14336×4096] | 117 MB | 0.64 ms | 0.72 ms |
| up | [14336×4096] | 117 MB | 0.64 ms | 0.72 ms |
| down | [4096×14336] | 117 MB | 0.62 ms | ~0.7 ms |
| q,o | [4096×4096] | 33.6 MB | 0.27 ms | ~0.3 ms |
| k,v | [1024×4096] | 8.4 MB | 0.12 ms | 0.13 ms |
| → FFN (gate+up+down) is ~3× the bytes of attention proj (qkv+o), so **FFN dominates | ||||
| the budget** (~60ms of the ~90ms matvec). Optimize FFN-shaped GEMVs first. |
7. Levers ruled OUT (don't revisit — measured)
- CPU AVX2 inner-dot: +3% (not compute-bound). CPU threads 4≈8 (8-vCPU cap, saturated).
- CPU lower quant (Q4_K==Q8): no gain — scalar K-quant dequant offsets the bytes.
- GPU sync-batching / FFN fusion: no gain — same bytes read (bandwidth, not sync, bound).
- f16-resident GPU: works + correct but reads 2× bytes → can't clear 15.
→ The ONLY lever is fewer VRAM bytes read at high bandwidth = optimized in-kernel
low-bit (Q8 no-loss → ~20-25 tps; Q4_K → ~30+, quality cost). See the handoff +
MESH_BUILD_DEPLOY_RUNBOOK.md.