forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

GKQ (rank-k SVD) is a NO-GO for Whisper — the weights are high-rank

distributed-inference/GKQ_WHISPER_HIGH_RANK_FINDING.md
forkjoin-ai/gnosis
Repo overview Open in workbench

GKQ (rank-k SVD) is a NO-GO for Whisper — the weights are high-rank

Date: 2026-05-23 Scope: GKQ spectral compression (quantize-whisper-weights.py, gkq_v2_format.py) applied to real OpenAI Whisper weights. Verdict: GKQ cannot compress Whisper without catastrophic quality loss. The density lever for Whisper is block quantization (q4/q8), not low-rank SVD.

TL;DR

GKQ approximates each weight matrix W by a rank-k SVD (W ≈ Uₖ·Σₖ·Vₖᵀ, default k=3) plus uint8 quantization. This only compresses well if the matrices are low-rank. Whisper's attention and FFN matrices (≈96% of params) are high-rank — their singular spectra are flat/diffuse — so the low-rank approximation throws away almost everything:

  • at the production default k=3: 94.9% mean relative Frobenius error (≈ random)
  • at k=64 (largest tested, 10 MB): still 64.8% mean error, cosine 0.75
  • no k ≤ 64 reaches even 20% error; faithful reconstruction needs k ≈ 384 (full rank → no compression, and larger than q8 due to uint8 overhead)

q8 block quantization (37.8 MB for whisper-tiny, WER 0.00 at exact parity) remains the floor. q4 (≈half of q8) is the next density step to validate.

Method

  • Model: whisper-tiny-hf-root/model.safetensors (167 tensors, 151 MB fp32; 67 are 2D matrices = the GKQ spectral path, 100 non-2D = dense passthrough).
  • Sweep rank k ∈ {1,2,3,4,8,16,32,64}, variance thresholds 0.999 / 0.9999.
  • For each matrix: GKQ quantize → dequantize (gkq_v2_format.reconstruct_spectral, signed offset-128 uint8) → relative Frobenius error ‖W−W'‖/‖W‖ and cosine sim.
  • Bench: bench_real_whisper_gkq.py. Independently cross-checked with a direct numpy.linalg.svd energy-fraction computation (numbers match to the decimal).

Singular-value energy captured by rank (independently verified)

Fraction of Σσ² retained by the top-k singular values:

tensor shape k=3 k=16 k=64 k=128 full
decoder.layers.0.self_attn.q_proj 384×384 5.8% 22.2% 62.4% 88.8% 100% (r=384)
decoder.layers.0.fc1 1536×384 3.9% 14.5% 41.8% 66.0% 100% (r=384)
encoder.layers.2.self_attn.out_proj 384×384 4.2% 18.4% 52.1% 78.0% 100% (r=384)
decoder.embed_positions 448×384 71.2% 96.9% 99.3% 99.7% 100%
decoder.embed_tokens 51865×384 88.7% 89.9% 92.3% 94.7% 100%

Effective rank for 99.9% energy across all 67 matrices: mean 327, median 328, p95 379, max 384 (= full). Only the positional embedding is meaningfully low-rank; attention/FFN are not.

GKQ reconstruction error vs size (whisper-tiny, all 67 matrices)

k GKQ size vs fp32 mean rel-err p95 cosine
3 (default) 2.69 MB 56× 94.9% 98.0% 0.27
8 3.31 MB 46× 91.2% 95.8% 0.38
16 4.29 MB 35× 86.3% 92.6% 0.48
32 6.25 MB 24× 78.2% 87.3% 0.60
64 10.17 MB 15× 64.8% 77.8% 0.75

q8 baseline: 37.8 MB, WER 0.00. GKQ at k=64 is smaller (10 MB) but ~65% error = garbage inference. The variance-threshold knob is inert here (vt_rank > k for all tested k).

Per-type mean error at k=3 / k=64: attention 96.9% / 64.0%, FFN 96.8% / 74.1%, embeddings 54.1% / 27.2%. Even embeddings stay lossy.

Why

Low-rank factorization assumes redundancy (a few directions explain the matrix). Trained transformer attention/FFN weights spread information across nearly all singular directions — the spectrum is approximately flat — so any aggressive rank truncation is near-random. This is a property of the architecture, not of this particular checkpoint or model size.

The synthetic-weights trap

The repo previously shipped rknots/whisper-base-gkq.knot (~750 KB), which made GKQ look viable. It was generated from generate_synthetic_whisper_weights() — random weights, which are also high-rank, so GKQ produced a small file that was never run through real inference. The tiny size was mistaken for success. Lesson: validate any compression scheme on REAL weights with an end-to-end quality gate (WER), never on synthetic tensors or reconstruction error alone in isolation.

Implication

  • Do not pursue SVD / low-rank compression for Whisper (or transformer weights generally). It is architecturally a dead end here.
  • The density lever is block quantization: q8 (proven, WER 0.00, 4× vs fp32) and q4 (GGUF-style 4-bit + per-block scale, ≈8× vs fp32, ≈half of q8) — this is what the bitwise gguf→knot pipeline does. q4 needs its own WER validation (small models are quant-sensitive).
  • For models too big for the browser even at q4 (medium/large-v3), run on the mesh via the gnosis native backend (fp32/q-block), not via spectral knots.

Reproduce

cd open-source/gnosis/distributed-inference
python3 bench_real_whisper_gkq.py            # full rank sweep on real whisper-tiny
# quick independent check (energy fraction by rank for one matrix):
python3 - <<'PY'
import json, struct, numpy as np
with open("../../../whisper-tiny-hf-root/model.safetensors","rb") as f:
    n=struct.unpack('<Q',f.read(8))[0]; h=json.loads(f.read(n)); base=8+n
    m=h["model.decoder.layers.0.fc1.weight"]; s,e=m['data_offsets']
    f.seek(base+s); W=np.frombuffer(f.read(e-s),np.float32).reshape(m['shape'])
sv=np.linalg.svd(W,compute_uv=False); en=np.cumsum(sv**2)/np.sum(sv**2)
print("fc1 energy@ k3/k64/full:", round(en[2]*100,1), round(en[63]*100,1), 100)
PY