forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

Sovereign IQ Encoder — Plan (mistral-7b first)

distributed-inference/SOVEREIGN_IQ_ENCODER_PLAN.md
forkjoin-ai/gnosis
Repo overview Open in workbench

Sovereign IQ Encoder — Plan (mistral-7b first)

Decision (user, 2026-07-16, locked): own the IQ-format ENCODE step so no crucial model depends on upstream llama.cpp. Prompted by mistral's IQ3 breakage being an upstream regression we can't reach. See memory sovereign-iq-encoder.

Why (diagnosis, already conclusive — no re-ladder needed)

  • Mistral IQ3 is wrecked: wikitext ppl Q4_K_M 6.42 · IQ3_M 13.76 · IQ3_XXS 15.51 · IQ2_S 13.91 · IQ2_XS 7.71 (healthy at LOWER bpw → non-monotonic → BUG, not cliff). Source: IQ_ROLLOUT_LOG.md wake-up 329 (build 7ecd2861, llama.cpp 3455882).
  • llama.cpp discussion #5866: same model+quant was ppl ~6.78 in early-2024 → a correct encoder is known-achievable; the defect is architecture-specific (qwen IQ3_XXS is healthy at 8.25). Owning encode removes this silent per-model risk.

What already exists (recon 2026-07-16, verified)

Piece Where
Bit-exact IQ3_XXS/IQ3_S/IQ2_S block layout + grid + sign tables src/iq_dequant.rs (IQ3XXS 98B; IQ3XXS_GRID[256], KSIGNS_IQ2XS, KMASK_IQ2XS)
Round-trip oracle (dequant fns + unit tests) src/iq_dequant.rs:159-291
Reference encoder algorithm (readable, vendored) scripts/fine-tuning/llama.cpp/ggml/src/ggml-quants.c:3866 quantize_row_iq3_xxs_impl
imatrix-weighted least-squares scale solve (== ggml sumqx/sumq2) src/kaiju_format.rs encode_bulk_row Fisher vertex m*
NATIVE imatrix capture (diagonal E[x²], no upstream) src/bin/kaiju-capture-activations.rs + kaiju_sketch.rs SketchEntry.diag
GPTQ error-feedback (optional, beyond ggml) src/kaiju_format.rs v3 block_solve/fold_feedback
KNOT writer that stamps a per-tensor type string kaiju-knot-driver.py (swap type:"KAIJU1" → "IQ3_XXS")

What must be built (small, well-bounded)

  1. Codebook nearest-neighbor for the 256-entry IQ3_XXS grid (brute-force NN over 4×int8 tuples, or regenerate ggml's kmap_q3xs/kneighbours from IQ3XXS_GRID).
  2. quantize_row_iq3_xxs Rust port → new src/iq_encode.rs mirroring iq_dequant.rs: per-32 sign extraction with even-parity enforcement (flip min-cost element), is∈[-15,15] scale search maximizing sumqx²/sumq2, scale=sumqx/sumq2, block super-scale d = max_scale/31 * 1.0125, nibble pack into scales_and_signs. Reuse KAIJU's scale-solve; imatrix weight w = H[i]*sqrt(sigma2 + x[i]²).
  3. Bit/quality test harness: structural (dequant(encode(W)) round-trips) + differential vs a KNOWN-GOOD ggml reference + the ppl quality gate.

Acceptance criteria (NOT ggml bit-exactness — ggml may be buggy)

  • Structural: emitted blocks are valid 98-byte IQ3_XXS; dequantize_iq3xxs_row round-trips within tolerance. (Existing kernel + tests are the oracle.)
  • Quality (the real gate): mistral-7b IQ3_XXS ppl comes back HEALTHY — monotonic vs IQ2 rungs, ~8-class not 13–15 — via scripts/eval-sweep.sh + native-pipeline-smoke Paris gate. This is the pass/fail bar.

Sequence

  • Step A — resolve the oracle (1 cheap build, gates the rest): quantize mistral-7b IQ3_XXS with the VENDORED tree (a827459b), measure ppl.
    • Healthy (~8-class) → vendored tree is PRE-regression → (i) immediate sovereign mistral unblock via our own in-repo quantizer, (ii) trustworthy bit-exact oracle for the Rust port. Best case.
    • Regressed (13–15) → vendored tree also carries the bug → source the pre-regression IQ3 impl (early-2024 llama.cpp) as the algorithm reference; vendored tree is only a STRUCTURAL oracle, quality comes from the correct algorithm + ppl gate.
  • Step B — native imatrix: run kaiju-capture-activations on mistral over the wikitext calibration file → SketchEntry.diag per tensor tap. Sovereign imatrix.
  • Step C — Rust encoder: build src/iq_encode.rs (items 1–2), test harness (item 3).
  • Step D — produce mistral knot: KNOT writer path (option b), type:"IQ3_XXS".
  • Step E — validate: eval-sweep + native Paris gate; ppl must be healthy. Ship the mistral IQ3_XXS knot; clear the mistral embargo in IQ_ROLLOUT_LOG.md.
  • Step F — fleet migration: once proven, migrate other models off llama.cpp encode.

Relationship to the qwen fleet rollout

Independent, parallel. The qwen fleet ships on VALIDATED llama.cpp IQ3_XXS (qwen is healthy). This track only replaces ENCODE, and only where it earns it (mistral first). Do NOT block the fleet on the encoder.

Output-path decision: emit .knot directly (recon option b)

Less work + fully sovereign. Reuse kaiju-knot-driver.py's assemble-from-base-knot pattern; overwrite 2D attn/ffn tensors with IQ3_XXS blocks under the real type string; copy embeddings/norms/lm_head verbatim; existing iq_dequant.rs kernels serve them. GGUF container only needed for external llama.cpp interop — not our serving path.