KAIJU1 native perplexity (wikitext-2 test, ctx=128)

Harness: src/bin/kaiju-ppl.rs (all-position logits via new NativeLlamaPipeline::lm_head_all_positions, src/model.rs). Runs the exact native-pipeline-smoke forward path (mmap knot → embed → 28 layers → lm_head) and scores with the standard llama.cpp perplexity() convention.

Result

• KAIJU1 (qwen2.5-7b, 3.11 GB knot, ~3.07 bpw) — native PPL = 19.78 @ ctx=128
• Converged, stable band 19.6–21.2 from chunk 9 onward.
• Scored tokens: 1008 (16 non-overlapping 128-token chunks, positions [64,127) of each — llama.cpp first = n_ctx/2 second-half scoring).
• Forward cost: ~90 s/chunk (no cross-chunk prepack cache in this build; each 128-token window re-dequants the full model through the Leech decoder).

Re-measured (2026-07-10, bounded 4-chunk slice)

Regenerated wiki.test.ids.u32 from scratch (tokenize.py, committed) after the prior tokens file was lost — Qwen2.5 BPE, add_special_tokens=False, over wikitext-2-raw/wiki.test.raw. Token count = 299,078, matching the original full-file count exactly (tokenizer/settings confirmed identical).

• KAIJU1 native PPL = 13.72 @ ctx=128, chunks=4, scored_tokens=252 (mean_nll 2.6188, wall 421.7s, ~105s/chunk).
• Running PPL climbed 6.71 → 12.43 → 12.86 → 13.72 across the 4 chunks — still on the upward trajectory toward the full-16-chunk 19.78 (which only settles into the 19.6–21.2 band from chunk 9 on). A 4-chunk head slice is therefore a lower partial average, not a discrepancy.

Slice (reproducible)

• Corpus: wikitext-2-raw/wiki.test.raw (ggml-org CI zip, the standard llama.cpp wikitext-2 test split).
• Tokenizer: apps/skymesh/src/ask/qwen2.5-tokenizer.json (Qwen2.5 BPE, add_special_tokens=False → no BOS, matching the qwen2.5 GGUF add_bos_token=false). Full file = 299,078 tokens.
• Slice measured: first 16 chunks = tokens [0, 2048).
• Convention: n_ctx=128, non-overlapping stride=128, first=64, predict token[start+pos+1] from logits at pos for pos in [64,127), PPL = exp(mean_nll).

Reproduce:

# 1. tokenize (Python tokenizers 0.22)
python3 tokenize.py apps/skymesh/src/ask/qwen2.5-tokenizer.json \
    wiki.test.raw wiki.test.ids.u32
# 2. measure
cargo build --release --bin kaiju-ppl
target/release/kaiju-ppl scratch-kaiju/qwen2.5-7b-kaiju1.knot \
    --tokens-file wiki.test.ids.u32 --ctx 128 --max-chunks 16

Comparability caveat

The IQ-ladder ledger baselines were taken with llama-perplexity -c 512 (scripts/eval-sweep.sh) on a fixed slice:

The native pipeline is hard-capped at ctx=128 (BATCH_CAPACITY / KV_MAX_SEQ_LEN = 128), so 19.78 is a ctx=128 number and is NOT directly comparable to those ctx=512 values. At ctx=128 the scored positions (64–127) see only 64–127 tokens of context vs 256–511 at ctx=512, which inflates ppl substantially for any model. No matched ctx=128 baseline anchor was run: no qwen2.5-7b GGUF and no built llama-perplexity are present locally (only vocab-only ggufs), and per instructions no multi-GB gguf was downloaded. To get an apples-to-apples verdict, run one baseline knot (IQ3_XXS/Q4_K) through this same kaiju-ppl harness at ctx=128 on the same slice.

ctx=512 measurement (2026-07-11) — now apples-to-apples with the ledger

The ctx=128 hard cap was the harness limit, not a model limit: the KV cache already defaults to 8192 and only BATCH_CAPACITY (the batched-prefill window + batch_* scratch) pinned the window to 128. Made that native capacity env-configurable (MOONSHINE_BATCH_CAPACITY, default 128, floor-clamped so the mesh boot budget can never shrink; wasm unchanged) and routed prefill_capacity, max_batch_tokens, the seq_len <= cap prefill guards, and the batch buffers through it (src/model.rs). Rebuilt kaiju-ppl and scored at ctx=512, the exact window the ledger baselines use (llama-perplexity -c 512).

MOONSHINE_BATCH_CAPACITY=512 target/release/kaiju-ppl \
    scratch-kaiju/qwen2.5-7b-kaiju1.knot \
    --tokens-file wiki.test.ids.u32 --ctx 512 --max-chunks 60

Result — KAIJU1 (qwen2.5-7b, 3.11 GB knot, ~3.07 bpw) native PPL = 11.7643 @ ctx=512.

• Slice: 60 non-overlapping 512-token chunks = tokens [0, 30720) of the same 299,078-token wiki.test corpus; first=256, score positions [256, 511) → 255 predictions/chunk × 60 = 15,300 scored tokens.
• mean_nll = 2.465066, wall 21,366 s (5.94 h, 356.1 s/chunk), nice -n 10.
• Convergence: running PPL peaked at chunk 13 (14.32), fell to a chunk-33 trough (10.46), then oscillated in a ~10.5–11.8 band from chunk 33 to 60 (<1% chunk-to-chunk drift over the last ~20 chunks). Converged, not a partial.
• Sanity: 11.76 @ ctx=512 < 19.78 @ ctx=128 (−40.5%) — longer context lowers ppl exactly as expected. The ctx=512 number is the ledger-comparable one.

Apples-to-apples verdict

KAIJU1 does NOT beat IQ3_XXS at equal size — it is +42.6% worse, and worse even than IQ2_M despite being 0.34 GB larger. This directly falsifies the ledger's tensor-excess→ppl prediction (that KAIJU1 @ 3.07 bpw would land below IQ3_XXS @ 3.06 bpw) and confirms the earlier ctx=128 "v1 FAILS the quality gate" anchor (19.78, +58% vs baseline) on the now-comparable ctx=512 axis. The excess→model-ppl bridge did not transfer; KAIJU1 v1 is distributionally well below the IQ ladder at its size point.

Comparability caveat: the ledger baselines average the full wiki.test (584 chunks); this run is the first 60 chunks (30,720 tokens, 15,300 scored, ~10.3% of the corpus). The running value is plateaued, so 11.76 is a solid estimate of the corpus mean — and the +42.6% gap to IQ3_XXS is far larger than any residual head/tail-slice bias could close.