forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

KAIJU1 — Leech-24 bulk + top-H side-band weight format

distributed-inference/KAIJU1_FORMAT.md
forkjoin-ai/gnosis
Repo overview Open in workbench

KAIJU1 — Leech-24 bulk + top-H side-band weight format

Tensor-level quantization format for .knot models. Carried as ordinary knot tensors with TOC type: "KAIJU1" — no container change (the kn0t lesson: the knot already carries any type string; dump-knot.ts keeps working).

Why this exact shape — every choice is a measured number

The 2026-07-08 measurement campaign (IQ_ROLLOUT_LOG.md, journal wake-ups 6–22) established, on real qwen2.5-7b tensors with real imatrix H:

  1. 91.7% of practical IQ2_S's weighted excess lives in the top decile of H-channels (validated: relMSE 12.35%, no artifact). Mechanism: tiny grid dynamic range clips outlier weights exactly where the function listens hardest. ⇒ The format's centerpiece is the side-band, not the lattice: the top-H channels get a high-precision lane.
  2. Practical IQ2_S sits ~4× above the ideal-lattice floor at matched coarse rates (weighted excess 22.38 vs 5.91 @1.4σ) ⇒ ~1+ bits/dim of recoverable headroom exists once clipping is removed.
  3. Λ24 beats ideal E8 by 8.4% (white, G=0.065745 measured) to 13.4% (real weights + real H) and never inverts ⇒ the bulk lane uses the Leech lattice — free, sovereign, Lean-certified (Gnosis.LeechNearestPoint, Gnosis.HopeJarKaiju), decoder proven Voronoi-exact over all 196,560 minimal vectors.
  4. Per-TENSOR protection was measured null-to-harmful (IQ2_S-prot 9.436 > plain 9.323) ⇒ the side-band granularity is per-CHANNEL, chosen by measured H, never by tensor name.

Formal anchors: Gnosis.GodFormulaQuant (allocation law: high-H coordinates get the bits — the side-band IS high_H_gets_small_error at format scale); Gnosis.TritonQuantizationInversion (trit packing fact 3^5 = 243 ≤ 256 used by the bulk residual lane; quantization-beats- truncation on flat spectra); Gnosis.HopeJarKaiju (Leech tier capacity = |Co₀|, the naming SSOT).

Per-tensor layout

For a rows×cols weight tensor (cols = input channels, matching imatrix H):

[KaijuHeader]
  magic          u32   'KJU1'
  rows           u32
  cols           u32
  hi_count       u32   number of side-band channels (default ⌈0.10·cols⌉)
  bulk_cols      u32   cols − hi_count, padded ↓ to multiple of 24 (tail
                       channels beyond the pad go to the side-band too, so
                       nothing is dropped)
  step_num/den   u32×2 global bulk step as a rational × per-row scale
[HiIndex]        u32 × hi_count      ascending column ids of side-band
                                     channels (bulk permutation is derived
                                     from this — no extra storage)
[HiBand]         per hi channel: f16 scale + i8 × rows   (≈8 bpw lane)
[BulkRows]       per row:
  row_scale      f16
  per 24-block:  blk_scale_q4 (4b, K-quant style multiplier on row_scale)
                 + LeechCode (see below)

LeechCode — bounded coding of the nearest Λ24 point (56 bits/block)

The Conway integer coordinates of nearest_leech(y) decompose per the even/odd Golay-coset characterization (leech_quantizer.rs::is_leech_point):

parity m           1 bit
Golay codeword     12 bits  (index into the 4096 codewords, generator-order)
coord residuals    24 × ternary {-1,0,+1} → 8 bytes trit-packed (3^5/byte)
                   = z-part of each coordinate after removing m + coset
                   contribution, clamped to the ±1 box
sum-fix index      5 bits   (which coordinate absorbed the mod-8/mod-4 sum
                   correction; 0x1F = none)
padding            → 56 bits total = 2.33 bpw bulk lane

No-clipping guarantee replaced by graceful degradation: residuals outside the ternary box are projected to the nearest representable point. This is safe by construction of the partition: the side-band has already removed the channels where projection error × H is expensive (measured: 92% of the damage). Bulk channels are low-H by definition — projection there is the cheap kind of error. This is the format-level embodiment of GodFormulaQuant.zero_H_error_is_free / high_H_gets_small_error.

Rate budget (cols-amortized, hi_frac = 0.10)

side-band:  0.10 × (8 + 16/rows)          ≈ 0.80 bpw
bulk:       0.90 × (56/24 + 4/24)         ≈ 2.25 bpw
row scales: 16/cols                        ≈ negligible
TOTAL                                      ≈ 3.05 bpw v1

v1 target: ≤3.1 bpw with weighted excess ≪ IQ2_S's (the napkin from the measured gap: bulk excess at these steps is 3-4× lower than IQ2_S's clipped excess, and the side-band nearly eliminates the top-decile term that was 91.7% of the total). Success criterion at the gate: beat IQ3_XXS quality (ppl ≤8.25) at ≤ IQ2_M size (≤2.8GB) on qwen2.5-7b. v2 levers if the gate demands them: entropy-coded Golay indices (measured usage is skewed), hi_frac tuning, 5-bit side-band.

Encode / decode

  • Encoder: src/kaiju_format.rs::encode_kaiju1(tensor_f32, h, rows, cols) — partition by H (needs the model's imatrix, which the fleet pipeline already produces per model), side-band pack, per-block scale fit + nearest_leech (4.5 ms/pt: ~2-3 min/7B-tensor-set on 32 cloud cores; encode-once). A hexacode fast decoder is a wanted optimization, not a blocker.
  • Decoder: dequantize_kaiju1_row(out, bytes, row, cols) — same signature family as iq_dequant.rs, slots into the existing QType dispatch (the 18 sites wired 2026-07-08 give the integration map).
  • Gate: identical to the IQ2_M chain — encode qwen2.5-7b → knot → native Paris + eval sweep vs the IQ2_M and IQ3_XXS anchors.

Provenance

Measured inputs: builds 8b6ca582 (ladder), 3818de86 (recovery), b816ef4a + validation rerun (gap + mechanism), leech test suite (G=0.065745, exhaustive shell decode). All numbers in IQ_ROLLOUT_LOG.md.