KAIJU v3 Implementation Plan — GPTQ-lite over Λ24

Status: PLANNED 2026-07-15 (not started). Authored from the 4-agent exam of build 52481d88 (fisher=8 regression) + theory audit of the diagonal-ledger/ppl inversion. Executing session: read this whole doc; every design decision below is anchored to verified source facts.

Why v3 (context)

KAIJU1 measured 11.5× LOWER diagonal-H weighted excess than IQ2_S on identical rows yet full-model wikitext ppl 11.76 vs IQ3_XXS 8.251 at equal size (ctx=512, native harness — harness audited SOUND). Theory verdict: the diagonal-Fisher ledger omits the off-diagonal quadratic term Σ_{i≠j} S_ij ΔW_i ΔW_j (S=E[xxᵀ]); KAIJU's error is block-correlated (shared per-block scale + joint 24-dim lattice decision, aligned with trained-row directions) which makes that term large and positive, while IQ's near-white per-coordinate error self-averages it to ~0. In-repo proof: the fisher step search provably lowered the diagonal objective yet catastrophically worsened quality (0.5b Paris rank 1→39). The closed-form fisher rewrite (7010015c) additionally has a code-step mismatch defect (re-snaps the 4-bit multiplier WITHOUT re-encoding the lattice code at the new step — kaiju_format.rs:339-361); treat fisher_steps>0 as broken, K=0 is the trusted legacy path.

v3 = sequential 24-column block encode with error feedback against the FULL covariance Ŝ (GPTQ-lite): after committing each block, project its error through Ŝ and fold compensation into not-yet-encoded columns, minimizing ΔwᵀŜΔw rather than the diagonal proxy.

Ground-truth facts verified in source (anchor every decision)

• src/leech_quantizer.rs:867 nearest_leech — exhaustive MLD over both Golay coset halves (2 × 4096 codewords × 24 coords; decode_leech_even/odd at 758/813). NO hexacode anywhere in repo. Golay machinery a hexacode decoder needs (sorted codeword table, octads, weight enumerator, is_leech_point, NSM-band harness test_c at 1332) all present = differential-test oracle.
• src/kaiju_format.rs encode_block (130): one nearest_leech call/block; reconstruction strictly linear: rec = step·Ĉ, Ĉ = decode_block(code, 1.0). Decoder ignores byte1 bits 5–6 (line 183: cw_idx = (code[0]>>1) | ((code[1]&0x1F)<<7), bit7 = flip) — 2 reserved bits/block pass through the shipped decoder untouched.
• Serving dispatch purely by knot TOC type string → QType::KAIJU1 (model.rs:587); knot metadata is serde_json::Value (loader.rs:1521) — tolerant of extra JSON keys.
• Activation tap points in NativeLlamaPipeline (buffers private to model.rs, so taps live there): batch_xb post-attn-norm ~10029–10044 (input to q/k/v); batch_xb pre-O-matmul ~10389–10409 (input to attn_output); batch_xb at forward_range_ffn_gate_chunk entry 10484/10533 (input to gate/up); batch_hb post-swiglu at forward_range_ffn_down_chunk 10772–10786 (input to ffn_down). Two capture patterns to copy: env-gated global sink (src/residual_capture.rs, GNOSIS_RESIDUAL_CAPTURE) and buffered per-layer capture + PCA fit (standing_wave_capture, model.rs:3637–3700).
• iq_dequant.rs has IQ2_S and IQ3_S only — NO IQ3_XXS dequant (G0 must handle via python gguf.quants.dequantize in a driver step).
• Ops wiring: cloudbuild-kaiju-encode.yaml (MONOREPO ROOT), submit wrapper scripts/kaiju-zoo-submit.sh (dry-run default; substitutions derived from monster-swarm/qspecs/<id>.qspec.json via scripts/kaiju-gate-params.py); watcher LaunchAgent scripts/com.forkjoin.kaiju-zoo-watcher.plist. Ship step is UNCONDITIONAL (yaml 252–272) — always-ship footgun is live; _PPL_MAX/_KL_MAX default empty. This destroyed the legacy 11.76 knot (overwritten by 52481d88).
• gnosis-math/Gnosis/HopeJarKaiju.lean is a Co₀-capacity certificate only — no decoder structure to mirror. Lean anchor for the decoder is Gnosis.LeechNearestPoint (leech_quantizer.rs:54); hexacode certificate = new work, OUT OF SCOPE for v3.
• 0.5b assets: paris.qspec.json (prompt [785,6722,315,9625,374] → expect 12095), f16+imatrix+base knot present per KAIJU_ZOO_PLAN.md. Whether a 0.5b KAIJU1 knot shipped is unconfirmed — Phase 1 probes R2 and runs a cheap K=0 legacy encode if absent.
• Fisher8 forensic artifacts preserved at ~/.edgework/models/ qwen2.5-7b-kaiju1-fisher8.{knot,telemetry.json,gate.log,ctx512-ppl-partial.log} — the knot is the strongest known "lower diagonal excess, worse quality" instance; key G1 input.

0. Architecture decision: qtype and version marker

v3 keeps the KAIJU1 wire format and QType::KAIJU1 unchanged. No new qtype. Version carried out-of-band (knot meta JSON + R2 key + telemetry) plus optional in-band watermark in a decoder-ignored reserved bit.

1. GPTQ error feedback is encoder-only: changes WHICH lattice point and WHICH 4-bit multiplier are stored, never the layout. Frozen decoder reconstructs step·Ĉ identically; all shipped serving binaries serve v3 knots with zero changes.
2. A new magic (KJU3) would force Kaiju1Layout::parse changes in every deployed decoder for no informational gain.
3. Version marker, three layers:
   • Knot meta JSON (authoritative): driver writes top-level "kaiju": {"encoder":"v3-gptq-leech","sketch_id":"<sha>", "decoder":"hexacode","step_search":4} into new_meta (kaiju-knot-driver.py:155). Old loaders ignore extra keys.
   • R2 key: models/<id>-kaiju3-r<N>.knot — new key per attempt, monotone -rN, NEVER reuse _OUT_KEY (destroyed-artifact footgun).
   • In-band watermark (free): set byte1 bit6 = 1 in every v3 block; shipped decoders mask it off. 0-bit cost.

1. Ŝ definition, capture, storage

Ŝ per tap = diag(E[x²]) + low-rank Nyström sketch of E[xxᵀ], diagonal exact: Ŝ = D_resid + U Λ Uᵀ, D_resid = max(diag(E[x²]) − diag(UΛUᵀ), ε), K=32 (G1 ablates K ∈ {16,32,64,128}).

Tap families (one Ŝ per (layer, family), shared by consumers):

Accumulation (streaming, single pass, no raw-activation files):

• hidden-dim taps: exact f64 Gram, d²×8 = 6.4 MB/tap (0.5b) → ~460 MB total for 24 layers × 3 taps. For 7B (103 MB/tap × 72 taps = 7.4 GB) switch to f32 or sketch path.
• ffn_down_in taps: one-pass Nyström — fixed Gaussian Ω ∈ R^{d×R}, R=128, accumulate Y += x·(xᵀΩ) (d×R f64 ≈ 5 MB/layer); top-K eigenpairs from Y(ΩᵀY)^†Yᵀ.
• Eigensolve: small symmetric (≤R×R) — plain Jacobi in the new module (standing-wave PCA fit at model.rs:3678 is the in-repo precedent).
• Calibration: ~100k wikitext-TRAIN tokens (disjoint from wiki.test.ids.u32), ctx=512 (MOONSHINE_BATCH_CAPACITY=512), position stride 2 → ~50k samples/tap. Built-in correctness check: recovered diag must rank-correlate ≈1 with the shipped imatrix H (same E[a²]).

Sidecar format .sket (little-endian):

"KSK1" u32 | version u32 | model_id_len u16 + utf8 | n_entries u32
per entry:
  name_len u16 + utf8            # tap name, e.g. "blk.0.ffn_down_in"
  dim u32 | k u32 | n_samples u64
  f32 diag[dim]                  # exact E[x_i²]
  f32 eigval[k]                  # λ (uncentered second moment)
  f32 eigvec[k*dim]              # row-major orthonormal U

0.5b total ≈ 33 MB. Ship to R2 models/sketches/<id>-<sha8>.sket; sha recorded in knot meta.

2. v3 encoder algorithm (GPTQ-lite over Λ24)

Per row, per tensor, Ŝ restricted to the BULK column permutation (side-band columns removed from D and U before Cholesky logic — they carry near-exact 8-bit values, quantized first, their small error optionally folded once into bulk targets):

targets t = row bulk values (after side-band commit)
for each 24-block B in order (natural bulk order; act-order = extension):
    step search: for q in {q0-1, q0, q0+1, q0+2} (clamped 0..15):
        code_q  = nearest_leech_fast(t_B / step(q) * √8)   # re-encode at EVERY
                  # candidate step — fixes the 7010015c defect by construction
        E(q)    = (t_B − step(q)·Ĉ_q)ᵀ Ŝ_BB (t_B − step(q)·Ĉ_q)
    commit argmin q; e_B = t_B − step·Ĉ
    feedback (block Gauss elimination on Ŝ):
        g = Ŝ_BB⁻¹ e_B                # 24×24 solve, Ŝ_BB = D_B + U_B Λ U_Bᵀ
        t_rest −= U_rest Λ (U_Bᵀ g)   # O(d_rest·K); off-diag blocks of Ŝ are
                                      # exactly U_B Λ U_restᵀ (D diagonal)

Never materialize dense Ŝ (7B ffn d=18944 → 2.9 GB). Rows independent → rayon over rows preserved (feedback sequential only WITHIN a row). fisher_steps stays a dead legacy flag; v3 selected by new --v3 --sketch args; K=0 legacy remains byte-identical and untouched (regression test asserts).

Clamp interaction: feedback grows targets; track per-block clamp telemetry; cap compensation at (configurable) 2× block RMS; hard gate clamp ≤ 1.5× legacy encode of same tensor.

Phase-3 implementation corrections (2026-07-15, measured — see kaiju_format.rs FoldState): two defects in the literal pseudocode above, both empirically confirmed and fixed in the shipped encoder:

1. SIGN: with e_B = t_B − step·Ĉ the fold t_rest −= Ŝ_rB Ŝ_BB⁻¹ e_B ANTI-compensates (2-var check: committing e₁ = w₁−ŵ₁ must shift t₂ to w₂ + ρe₁; test (c) showed ~4× fullS blowup). Correct fold solves with e = rec − t.
2. FIXED-Λ FOLD DIVERGES: folding every block through the ORIGINAL U Λ Uᵀ re-corrects the same global eigendirections once per block — on real 7B attn_output rows with a λ-dominated sketch this blew fullS excess up 767×. The encoder instead does EXACT sequential block Gauss elimination in low-rank form: keep a K×K capacitance M (init Λ), fold through U M Uᵀ, Schur-update M ← M − M U_Bᵀ Ŝ̃_BB⁻¹ U_B M per committed block (side-band is the opening block). Same O(d_rest·K) fold + O(K²·24)/block. Result on the same rows: fullS 0.75× v1 (vs 0.76× step-search-only), trit clamp DOWN v1 0.025 → v3 0.010, comp-cap never engaged.

3. Encode-cost model

Per block: 4 nearest_leech calls (one per step candidate; feedback adds zero calls — Ĉ from decode_block(code,1.0), microseconds).

This is why G2 gates G3: exhaustive v3 on 7B is 42 h against the 24 h build timeout wall; hexacode makes even 70B (~10×) plausible later.

4. File-by-file change list

New files

1. src/leech_hexacode.rs — fast MLD Leech decoder.
   • Tables: (6,3,4) hexacode over GF(4) (64 codewords), MOG column mapping 24 coords → 6 GF(4) symbols; per-coordinate residue-class penalty tables for both even/odd halves (reuse nearest_in_residue congruence structure from decode_leech_even/odd).
   • pub fn nearest_leech_fast(x: [f64;24]) -> [i64;24] — Amrani–Be'ery–Vardy-family decode: hexacode-driven MLD of the Golay per half + existing least-metric Σ-parity repair; same output convention (Conway integer coords, min-norm-32).
   • PRAGMATIC FALLBACK same module: nearest_leech_pruned — exact MLD via branch-and-bound over the sorted codeword table with per-coordinate metric lower bounds + early abort (~10–50×, still exact). If hexacode slips, G2 passes on this at ~100–400 µs/pt (7B lattice ~4–8 h on 32 vCPU — viable).
   • Tests: differential vs nearest_leech (exact point equality or tied dist_sq within 1e-9) on ≥10⁶ SplitMix64 uniforms + ≥10⁵ real 0.5b weight blocks at encoder-realistic steps; is_leech_point on every output; test_c NSM harness re-pointed at fast decoder must land in 0.062–0.067; timing print.
2. src/kaiju_sketch.rs — .sket read/write, Gram/Nyström accumulators, Jacobi eigensolve, struct SketchEntry { diag, eigval, eigvec } with fn quad_form(&self, dw) -> f64 (= Σ D_resid·dw² + Σ λ(uᵀdw)²) and fn block_solve / fold_feedback helpers.
3. src/bin/kaiju-capture-activations.rs — loads a 0.5b knot (best available base quant; capture quality insensitive), replays the kaiju-ppl chunk-forward loop (embed_chunk → forward_range_*_chunk) over calibration tokens with taps enabled, accumulates, eigensolves, writes .sket. Args: <knot> --tokens-file --ctx 512 --max-tokens 100000 --stride 2 --k 32 --out <model>.sket.
4. src/bin/kaiju-sq-score.rs — G1 scorer: f16 tensors (f32-dumped by driver step), KAIJU1 knot, IQ baseline knot, imatrix, .sket; per tensor emits {diag_excess, fullS_excess} per artifact + model-wide sums + ordering verdict as one JSON line.
5. /Users/buley/Documents/Code/monorepo/cloudbuild-kaiju-v3-encode.yaml — clone of cloudbuild-kaiju-encode.yaml adding: fetch-sketch step (R2 _SKETCH_KEY), _KAIJU_V3='1', _STEP_SEARCH='4', _DECODER='hexacode', NON-EMPTY defaults for _PPL_MAX/_KL_MAX/_REF_KNOT_KEY (v3 builds must not inherit ship-always), _OUT_KEY validated to contain -kaiju3-.

Modified files 6. src/model.rs — four env-gated taps (pattern: residual_capture.rs OnceLock sink, zero-cost when GNOSIS_KAIJU_ACT_TAP unset). Insertion: after ~10044 (attn_in), before 10396 matvec (attn_out_in), before 10534 matvec (ffn_in), before 10773 matvec (ffn_down_in). Sink API: kaiju_sketch::tap(layer, family, &slice, seq_len, dim). Must be inert under amplituhedron replay — capture runs with hot path disabled (AMPLITUHEDRON_HOTPATH=0 / amplituhedron_hotpath_enabled gating at model.rs:9970). 7. src/kaiju_format.rs — add pub fn encode_kaiju3(w, rows, cols, h, hi_frac, sketch: &SketchEntry, step_search: usize, decoder: DecoderKind) -> (Vec<u8>, V3Telemetry) + encode_bulk_row_v3 per §2. encode_block gains variant taking pluggable fn(&[f64;24])->[i64;24] decoder + watermark bit. NO changes to decode_block / Kaiju1Layout / encode_kaiju1 (legacy stays byte-identical; regression test asserts). 8. src/bin/kaiju-encode-tensor.rs — new args --v3 --sketch <file> --sketch-entry <tap> --step-search 4 --decoder hexacode|pruned|exhaustive; telemetry gains {"v3":true,"sketch":"<sha8>","fullS_excess_pre/post":...}. 9. src/bin/leech-vs-e8-shootout.rs — G0: add --mode model-gap beside run_gap_mode (line 296): iterate all KAIJU tensors of a knot; per tensor compute diagonal excess for (a) KAIJU1 dequant rows (Kaiju1Layout) and (b) IQ3_XXS rows supplied as python-dequantized f32 files (no rust IQ3_XXS dequant — driver step uses gguf.quants.dequantize), vs f16 truth, with GGUF imatrix H (rust legacy parser at line 67 rejects GGUF — port the python reader or pre-convert to per-tensor .f32 H files in the driver step). Emits per-tensor table + model-wide sums. 10. scripts/kaiju-knot-driver.py — pass-through of v3/sketch args; insert the "kaiju" version dict into new_meta (line 155); sketch sha in telemetry.json. 11. src/lib.rs — register leech_hexacode, kaiju_sketch. 12. scripts/kaiju-zoo-submit.sh / scripts/kaiju-gate-params.py — --v3 flag switching config to cloudbuild-kaiju-v3-encode.yaml and _OUT_KEY=models/<id>-kaiju3-r<N>.knot (N auto-probed via the existing r2_present helper — never overwrite).

5. Phases, estimates, gates (each gates the next)

Phase 0 — G0 ledger-closure pre-check (dev ~1 day; run ~2–4 h cloud ≈$5) Item 9 + small cloudbuild (clone cloudbuild-kaiju-diagnose.yaml pattern): fetch f16 from GCS, KAIJU1 knot + imatrix from R2, IQ3_XXS gguf; python dequantizes IQ3_XXS per tensor; rust sweeps. DISK: run in Cloud Build (diskSizeGb: 150) — NOT locally (15.2 GB f16 vs ~5-8 GB free); HTTP-range per-tensor streaming (GGUF header offsets, ranged reads, ≤2 GB resident) documented as fallback. GATE G0 PASS: model-wide Σ diag-excess(KAIJU1) < diag-excess(IQ3_XXS) while ppl orders the other way — closes the ledger contradiction at model scale, licenses the off-diagonal hypothesis. If KAIJU diag excess is NOT lower model-wide, the story changes (some tensor family is diagonally bad) → fix targeting before any v3 work.

Phase 1 — G1 full-S discriminator (dev ~2–3 days; 0.5b capture ~1–2 h) Items 2, 3, 4, 6. Precheck: probe R2 for models/qwen2.5-0.5b-kaiju1.knot; if absent run trusted K=0 legacy encode via kaiju-zoo-submit.sh qwen2.5-0.5b --submit (cheap; also mints 0.5b ppl/Paris baselines). GATE G1 PASS (all): (a) sketch diag ≈ imatrix H (rank corr > 0.95 per tap); (b) on the tensors carrying most excess, fullS_excess(KAIJU1)/fullS_excess(IQ)

1 while diag ratio < 1 — ΔwᵀŜΔw reproduces the ppl ordering the diagonal inverts; (c) ordering stable K=32 vs K=128 (else raise K). IF G1 FAILS: v3's objective is unvalidated — STOP, re-audit (cheap kill-switch before encoder work). Fallbacks: per-block scale decorrelation / random sign-flip whitening of block error.

Phase 2 — G2 hexacode decoder (dev ~3–5 days; highest uncertainty) Item 1. Pruned exact decoder FIRST (1 day, guaranteed exact, unblocks at reduced speed), hexacode second. GATE G2 PASS: exact agreement with nearest_leech (identical point or equal dist_sq within 1e-9 Voronoi ties) on 10⁶ uniform + 10⁵ real-weight vectors; NSM ∈ (0.062, 0.067) on test_c; ≤50 µs/pt single-thread (pruned fallback ≤400 µs acceptable with re-derived §3 cost table). Bounded-distance only acceptable if excess-vs-exhaustive ≤ 1.02 on real-weight vectors.

Phase 3 — v3 encoder + 0.5b G3 (dev ~2–3 days; 0.5b encode+gates ~2–4 h) Items 5, 7, 8, 10, 11, 12. Local unit gates first: legacy-byte-identity; v3-vs-legacy fullS_excess strictly lower per tensor; clamp ≤1.5× legacy. GATE G3 (0.5b) PASS: native Paris argmax = 12095 (informational; rank-collapse = instant fail); kaiju-ppl ctx=512 with --ref-knot = 0.5b IQ baseline: v3 ppl < KAIJU1-v1 ppl (necessary) AND within --ppl-slack of equal-size IQ baseline (target); mean-KL below v1. Run as cloud build via v3 yaml (exercises the exact 7B path at toy scale).

Phase 4 — G3 (7B) (~$35, ~1 day wall) kaiju-zoo-submit.sh qwen2.5-7b --v3 --submit → v3 yaml with _REF_KNOT_KEY=models/qwen2.5-7b-iq2_m.knot, explicit _PPL_MAX/_KL_MAX, _OUT_KEY=models/qwen2.5-7b-kaiju3-r1.knot. Success = ppl materially closes 11.76 → 8.25 at ~3.07 bpw; record verdict in IQ_ROLLOUT_LOG.md.

Total: ~2 weeks dev + ~$45 cloud.

6. Risk register (top 5)

7. Sovereign tooling notes

• Builds: gcloud builds submit --config=<monorepo-root yaml> --project=neutral-418500 . pattern (cloudbuild-kaiju-encode/diagnose/gate yamls at monorepo root). Wrapper: scripts/kaiju-zoo-submit.sh (dry-run default, prereq HTTP-range probes, qspec-derived substitutions); unattended drain via com.forkjoin.kaiju-zoo-watcher.plist.
• Gates/data: monster-swarm/qspecs/*.qspec.json = argmax-gate SSOT (0.5b = paris.qspec.json); artifacts via R2 proxy https://edgework-web.taylorbuley.workers.dev/api/v1/r2/distributed-inference
  • upload-to-r2-mpu.py with R2_PROXY_TOKEN secret.
• a0 xr deploy: N/A — batch encode + offline measurement; frozen decoder = no fleet rollout.
• f16 truth: gs://neutral-418500-models/models//-f16.gguf (IQ-ladder is the sole feeder; do not re-mint).

Critical files

• src/leech_quantizer.rs (exhaustive decoder = G2 oracle; Golay tables reused)
• src/kaiju_format.rs (extend with encode_kaiju3; decoder FROZEN)
• src/model.rs (taps ~10044/~10395/~10533/~10772; QType dispatch)
• src/bin/leech-vs-e8-shootout.rs (G0 run_gap_mode extension)
• /Users/buley/Documents/Code/monorepo/cloudbuild-kaiju-encode.yaml (clone for v3 incl. ship-always fix)