forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

MESH_GEMMA4_12B_HANDOFF.md

distributed-inference/MESH_GEMMA4_12B_HANDOFF.md
forkjoin-ai/gnosis
Repo overview Open in workbench

DEFINITIVE BISECT (2026-06-22): it's a MATH bug, NOT GPU/precision

Ran gemma4-12b through THREE interventions on the fresh image (digest a40fa6e3), all garbage:

  1. f16-GPU (default): lm_head argmax = 236770 (high-vocab unused-token salad).
  2. f32-ACTIVATION GEMM (FLUX_GPU_F32_ACT=1, MIN_D=3000, cpu-overflow): logits SHIFTED (argmax → 111038) but STILL garbage. So f16-input rounding is a CONTRIBUTOR, not the root.
  3. PURE CPU (FLUX_GPUFIX=1 FLUX_GPUFIX_MIN_D=0, exact f64 accumulate, rev 00095-tjk): lm_head argmax = 236770 — IDENTICAL garbage to f16-GPU. CPU == f16-GPU == 236770 salad ⇒ the bug is in the gemma4 MATH (model_gemma4.rs forward) or the gemma4-12b knot config — NOT the GPU path and NOT precision. The prior-session "CPU produces Paris @ +16.3" claim DOES NOT REPRODUCE on this image (model_gemma4.rs has changed since, or that measurement was on a different state). lm_head consistently favors high-vocab unused tokens (~236xxx of 262144) → suspects: final RMSNorm, the TIED-embedding lm_head (gemma ties embed/unembed; vocab 262144), the sqrt(hidden) embedding scale, or the final logit softcap. STOP chasing GPU/precision. NEXT (model_gemma4.rs = LIVE WIP, owner's): add a CPU golden-diff of the gemma4 forward vs HF transformers modeling_gemma3/4 layer-by- layer (embed scale → each block → final norm → lm_head) to find the first diverging op. GNOSIS_LM_HEAD_FP32=1 already on, so lm_head precision is not it. Mistral spec-decode work unaffected; prod restored to 00072-rp2.

V-NORM FIX: necessary but NOT sufficient (2026-06-22)

Removed the spurious dense-Gemma3 V-projection unit-RMS norm (was gated on arch=="gemma4", should be e4b-only — verified vs HF dense Gemma3 has no v_norm, only Gemma3n does). Built + deployed (img cdd4d329, rev 00096-bxf). RESULT: lm_head STILL argmaxes 236761/236770/236764 — logits essentially UNCHANGED. So V-norm was a real arch bug but NOT the dominant corruption. KEY INSIGHT — the garbage is INVARIANT across ALL interventions tried: f16-GPU, f32-act GPU, pure-CPU exact-f64, AND V-norm removal. Invariance ⇒ the corruption is in a path none of those touched: the final-hidden→lm_head readout, OR an always-applied per-layer scale. PRIME SUSPECT: the layer_output_scale tensor — the knot is labeled arch="gemma4" SOLELY because it carries this tensor (bitwise/scripts/encode-llama-knot.ts:437-438). If the forward mis-applies it (or applies a Gemma3n-only scale to dense Gemma3, exactly like the V-norm bug), it corrupts the residual on every layer → high-norm-embedding-row argmax (236xxx via tied unembed), invariant to attention/precision. NEXT: audit layer_output_scale application in model_gemma4.rs vs HF/the working TS aether reference (gcp-layer-node.ts) — is it applied, where, with what value, and does dense Gemma3-12B even have it? Also re-check final RMSNorm gain + embed sqrt-scale on the SAME hidden the lm_head reads. Mistral spec-decode prod restored (00093-vgs).

★ ROOT CAUSE = TOKENIZER, not model math (2026-06-22) ★

Instrumented the full gemma4 forward (GEMMA4_DBG, rev 00097-wkt). The model MATH IS SANE:

  • [g4embed] embed_scale=61.97=√3840 ✓, post_embed_rms=1.21 ✓
  • [g4final] PRE-norm last-tok rms=0.16 max=6.0 — residual BOUNDED + sane at the final layer (no NaN, no balloon). The layer stack works. (V-norm fix was a real bug; layer_output_scale is LOAD-BEARING — removing it → full NaN; keep it ON.) THE BUG IS THE INPUT TOKENS. "The capital of France is" tokenizes to: 105, 247723, 111038, 887, 236820, 511, 245237, 31756, 245237, 1340, 245237, 41626, 245237... 105= ✓ but the rest is high-vocab garbage with 245237 REPEATING 4×. A correct gemma tokenization = ~15 low/mid-vocab tokens. The model faithfully processes garbage tokens → garbage logits in the SAME high-vocab range (236761/236770). This is why the salad was INVARIANT across f16-GPU / f32-act / pure-CPU / V-norm / layer-scale — none touch the input. gemma4-12b uses tokenizerFamily:'qwen2' (generic GGUF loader) on a gemma SentencePiece vocab (262144). The generic/qwen2 BPE path almost certainly mis-loads gemma's SentencePiece merges/scores. FIX HERE: the gemma4-12b-it-tokenizer.gguf loading path (SentencePiece vs BPE, merges, byte-fallback, the ▁ space handling). Verify by tokenizing a known string vs HF/aether gemma tokenizer. Model-side gemma4 fixes are DONE (V-norm gated to e4b; keep layer_output_scale ON). Prod = 00093-vgs.

STATUS (2026-06-22): CORRECTNESS CRACKED, serving/VRAM blocker remains

Two REAL correctness bugs found + fixed (verified at component level):

  1. V-norm: spurious dense-Gemma3 V-projection unit-RMS norm (was arch=="gemma4"-gated; now e4b-only). Real bug, fixed.
  2. ★ TOKENIZER (the root cause of the salad): gemma SentencePiece vocab was mis-loaded via the qwen2 GPT-2-byte-BPE path → spaces collapsed to Ġ=245237 (repeating), words hit no-▁ variants → garbage tokens IN → garbage logits OUT, invariant to all model fixes. Fixed by routing gemma4 to the proven hf-bpe-metaspace tokenizer path (gemma3-4b uses it). VERIFIED: "The capital of France is" → [669,5279,529,7001,563] = ▁The▁capital▁of▁France▁is (was 247723,111038,...,245237×4). Input is now CORRECT on the live mesh (g4embed shows token=7001=▁France). Also: layer_output_scale confirmed LOAD-BEARING (default flipped back ON; removing → NaN). END-TO-END "Paris" NOT yet confirmed — blocked by an OPERATIONAL VRAM issue, NOT correctness: gemma4-12b on the 24GB L4 OOMs uploading an f32 weight ("[cuda] f32 weight htod_sync_copy failed: CUDA_ERROR_OUT_OF_MEMORY — falling back to CPU kernel"), driven by GNOSIS_LM_HEAD_FP32=1 in the REGISTRY (4GB fp32 lm_head over 262144 vocab). Service-level GNOSIS_LM_HEAD_FP32=0 does NOT override (registry env wins). CPU-fallback lm_head per token → generation too slow → mesh proxy "socket hang up". NEXT (serving, not correctness): drop GNOSIS_LM_HEAD_FP32 from the gemma4-12b/31b registry entries (the fp32 lm_head was chasing the precision red-herring; the real bug was the tokenizer, so Q6K lm_head on GPU is fine) + rebuild → gemma4 fits VRAM, decodes on GPU, completes. Then verify Paris. Prod restored to mistral spec-decode 00093-vgs.

FINAL (2026-06-22): correctness DONE; gemma4-12b doesn't fit the L4 → needs Q4_K requant

Removing the fp32 lm_head cleared THAT OOM, but the f16 lm_head (Q6K 262144×3840 = 2013 MB) STILL OOMs: "f16 resident 18.98 GB / budget 21.0 GB" → CPU-fallback lm_head → too slow → empty/timeout. gemma4-12b at Q6K (~19GB weights) + the 2GB lm_head exceeds the L4's usable VRAM (~20GB after CUDA ctx + KV). Lowering FLUX_GPU_F16_BUDGET_GB (service-env override of the registry, supervisor.ts:100) did not visibly take in the time tried. This is a HARDWARE/quant capacity wall, NOT correctness. THE FIX: requant gemma4-12b to a Q4_K_M knot (~12-13GB → fits with lm_head on GPU, like the 31b's Q4_K) and publish to R2; then it serves fast on the L4 AND amortizes the mesh's Q4_K GEMV kernel. Alternatively raise the GPU. CORRECTNESS IS RESOLVED (tokenizer + V-norm fixed, math verified sane, input tokens verified correct on the live mesh). Prod = mistral spec-decode 00093-vgs.

SERVING SOLVED, but forward has ≥1 MORE bug (2026-06-22)

RAW Q4_K KERNEL WORKS (img + MESH_GPU_Q4K=1, rev 00108-z66): logs show "Q4K-KERNEL(decode) gemm [4096x3840]·[1x3840] (raw-resident, no f16 expansion)" — no OOM, no thrash, generation COMPLETES. The VRAM wall is broken and gemma4 AMORTIZES the Q4_K kernel (wired into gemm_rows_shared_quant: tokens==1 + MESH_GPU_Q4K → raw q4k_gemv/q6k_gemv before the f16 path). No gpucheck WRONG, no CPU fallback → kernel is accurate. BUT output is still incoherent: "thought-10-1thought- ." and the prosody baseline (argmax) OSCILLATES between SANE (45518) and GARBAGE (236761/236772) tokens — same 236xxx ghost seen on pure CPU. So with input tokens now CORRECT (tokenizer fixed) and kernel ACCURATE, the gemma4 FORWARD still produces partly-corrupted hidden states → there is ≥1 MORE forward bug beyond the V-norm fix. Also slow: 0.88 tok/s (Q6K lm_head over 262144 vocab per token + 12b — needs the batched/kernel-opt the same way mistral got, secondary to correctness). FIXES BANKED (all real + durable): (1) V-norm e4b-gate, (2) tokenizer SentencePiece→hf-bpe- metaspace, (3) raw Q4_K kernel routing + no-fp32-lm_head. REMAINING: the deploy-cycle approach is exhausted — the forward bug needs a LAYER-BY-LAYER CPU GOLDEN-DIFF vs HF transformers modeling_gemma3 with the SAME weights (embed → each block hidden → final norm → lm_head logits), comparing the first diverging op. Suspects given the 236xxx (high-norm tied-embed rows): the tied-embedding unembed orientation/scale, a per-block residual/norm detail, or the Q6K lm_head dequant. NOT more env/deploy guessing. Prod = mistral spec-decode 00093-vgs.

REPLACEMENT VERDICT (2026-06-23): quality YES, speed NO

gemma4-12b as a mistral-7b replacement on the L4:

  • CORRECTNESS: SOLVED. Forward probe-verified (▁Paris), "Paris" live, tokenizer root-caused+fixed (chatTemplate gemma4→gemma; <|turn|>/<|channel|> were out-of-vocab). Committed d0b1c32e + 1a30f938. Clean output = 1 build away (build blocked by unrelated workspace WIP: aeon-ux submodule deps lean-proof-contract[untracked]/shared-ui + polyglot tree-sitter 0.25 vs moonshine ^0.24).
  • SPEED: NOT competitive. ~1.5-1.8 tok/s (after the shared-x fix 8cf60844) vs mistral 13-16. Decode is FFN-bound; residency is fine (no thrash). Down-proj fixed (x-bandwidth → weight- bound). But gate/up (~305ms/token combined, summed over 48 layers) are Q4_K-DEQUANT-COMPUTE- bound: Q4_K superblock dequant ≈ 5x q8's per-weight work, AND 12B = 1.7x params → ~8-9x mistral-7B-Q8's per-token dequant work. Even with a gate/up kernel-math rewrite, realistic ceiling is low-single-digit tok/s (maybe 3-4), NOT 13-16. FUNDAMENTAL: a 12B Q4_K on one L4 cannot match a 7B Q8 on decode throughput.
  • THE CALL: gemma4-12b is a QUALITY-OVER-SPEED option (smarter model, ~8-10x slower), not a drop-in mistral replacement. Frame it via the quality-reporting grades: mistral=A(13-16 tps, no quality loss); gemma4-12b≈D/C on tps but higher model capability. Only worth swapping for tasks where 12B quality justifies ~2 tok/s. To make it viable: (a) gate/up Q4_K dequant kernel rewrite, AND/OR (b) inherit spec-decode (the mistral AAA lever) for gemma4, AND/OR (c) a bigger GPU. Mistral stays the speed default.