forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

Model Zoo — admission → live-serving reverification checklist

distributed-inference-host/MODEL_ZOO_CHECKLIST.md
forkjoin-ai/gnosis
Repo overview Open in workbench

Model Zoo — admission → live-serving reverification checklist

Authoritative roster = the monster-guard admission ledger (open-source/gnosis/monster-swarm/ADMITTED_MODELS.md) + its golden qspecs/<id>.qspec.json. monster-guard ADMITTED each model offline (Cloud Build probe feeding pre-tokenized prompt_tokens, checking the expect token). This checklist tracks live-lane reverification: does it actually serve correctly through the real path (mesh fat-station HTTP-Range knot load + the openai-server tokenizer/template, or the edge fleet)? Admission ≠ serveable — the gaps are real (llama-3.2-3b shipped with the wrong tokenizer; gemma3-4b is admitted yet SIGBUSes on the mesh loader).

Bar = MESH_QUALITY_REPORTING_FRAMEWORK.md: correctness gates everything (token divergence ⇒ grade F). Reverify column: ✅ confirmed on lane · 🔧 fix in flight · ⏳ admitted+config-correct, probe pending · ❌ broken on lane (root-caused) · 🔲 not wired to a live lane · ➖ not a mesh/text target. Lanes: edge (CF Workers CPU, $0) · mesh (gnosis-openai-mesh L4 swap-on-demand) · fleet (10-node distributed edge CPU) · gpu-render (scale-to-zero L4) · browser-fleet (ONNX in viewer).

A. Text / LLM — dense transformer (mesh lane)

Model monster-guard qspec (expect) Reverify Note
qwen2.5-0.5b ADMITTED Paris 12095 ✅ edge FastPath distributed-coordinator-stream, 3-tier residency
tinyllama-1.1b ADMITTED Paris 3681 (needs BOS) ✅ edge after EDGE_PREFILL 8→32; zephyr template
deepseek-r1-1.5b ADMITTED Paris 12095 ✅ mesh correct after ~82s cold swap; edge coord not deployed
qwen3-4b ADMITTED Paris 12095 ⏳ mesh load-confirmed (hidden 2560/36L/vocab 151936/rope 1e6, tokenizer=qwen2, ready 640ms). Shares the qwen2+qwen-chatml path proven live on qwen0.5b → high-confidence. Clean live token deferred (swap-thrash w/ live session).
qwen2.5-7b ADMITTED Paris 12095 ✅ mesh "Paris" via isolated tagged rev (62s incl. cold swap)
qwen3-8b ADMITTED Paris 12095 ⏳ mesh config-correct (same proven path); validate on isolated --no-traffic tagged rev

Validation constraint: the mesh is single-GPU swap-on-demand (maxScale=1). It can't serve a live session AND validate other models simultaneously — competing model requests thrash the swap. Validate swap-required models on a --no-traffic tagged revision (its own isolated instance, 2nd L4) so probing doesn't contend with live traffic; promote after. | mistral-7b | ADMITTED | Paris 6233 | ✅ mesh | 0.71s warm ≈ grade A; no-regression anchor | | llama-3.2-3b | ADMITTED | Paris 12366 | ✅ mesh | FIXED: llama2→llama3 template + llama→qwen2 tokenizer + deployed wrong digest (tag lag); now d2f2d516, 62ms warm, "Paris" | | loki-8b-erotica | ADMITTED | Paris 12366 | ✅ mesh | "Paris" via isolated tagged rev (7s); reference llama3 tokenizer config | | gemma3-4b-it | ADMITTED (norm-swap fix) | Paris 9079 | ❌ mesh | SIGBUS on knot LOAD (head_dim=256/262k tied-embed overruns Range-mmap in loader.rs); admitted offline but native loader crashes. Engine fix. | | gemma4-12b-it | ADMITTED | Paris 50429 | ⏳ mesh | Q6_K embed + lm_head_fp32; chat-template gate; mesh probe pending | | gemma4-31b-it | ADMITTED | Paris 9079 | 🔧 mesh | ADDED to registry (reuses 12b tokenizer GGUF) — smoke-test next rebuild. ⚠️ RISK: ~17GB Q4_K + 4GB fp32 lm_head near 24GB L4 ceiling (OOM?); gemma4 CUDA path unproven (admitted via Vulkan); gemma3 loader-SIGBUS precedent. | | gemma4-31b-it-heretic-ara | ADMITTED · heresy | Paris 9079 | 🔲 — | 18.7GB Q4_K; Vulkan-validated; not a mesh target | | llama-3.3-70b | ADMITTED | Paris 12366 | 🔲 fleet | 71.5GB knot; NO-GO interactive (≈250× qwen0.5b = min/token); batch-only or multi-GPU. No per-layer blobs seeded. | | qwen2.5-72b | ADMITTED | Paris 12095 | 🔲 fleet | 73.7GB; ⚠️ alias bug qwen2.5-72b→qwen2.5-7b would silently serve 7B. Same fleet path/verdict as 70B. |

B. Text / LLM — SSM / hybrid / recurrent (separate /generate lane)

Model monster-guard qspec Reverify Note
mamba-2.8b ADMITTED Paris 7785 🔲 BLOCKED mesh-eligible (fits L4, /generate self-heals) but needs mamba-2.8b-tokenizer.gguf (gpt-neox-20b 50k vocab) published to R2 — a build step, not a registry edit. ~690s cold.
falcon-mamba-7b ADMITTED Paris 6671 🔲 BLOCKED same: needs falcon-mamba-7b-tokenizer.gguf (Falcon ~65k vocab) published to R2. Fits L4.
rwkv7-2.9b ADMITTED · reject-control pending Paris 37138 🔲 BLOCKED×2 no tokenizer GGUF + RWKV World is a trie tokenizer the qwen2 BPE path can't represent → needs a new tokenizer family. Admission reject-control still owed.
jamba-1.5-mini OBJECT PRESENT · qspec pending — 🔲 INELIGIBLE 54GB knot ≫ 24GB L4 → can't share the mesh. Solo big-GPU or skip (same class as 70B/72B).
jamba-mini-1.7 INCOMPLETE — ➖ experts dropped — use jamba-1.5-mini instead
qwen35moe-heretic OBJECT MISSING — 🔲 Qwen35MoePipeline impl'd; knot 404 — rebuild/upload blocker
gemma-4-E4B-heretic OBJECT MISSING — 🔲 PLE+KV-share impl'd; full --capture-all re-encode owed

C. Multimodal (non-mesh lanes)

Model modality monster-guard Reverify Note
whisper-tiny ASR ADMITTED ➖ exact token match; same engine as mesh
qwen2.5-vl-3b / qwen2-vl vision-language ADMITTED 🔲 gpu/≥24GB ViT→256 visual tokens→NativeLlama; text-only mesh can't serve VL — GPU- / big-RAM-gated
glm-ocr OCR ADMITTED ✅ local LIVE locally via src/bin/ocr-server.rs (GLM-4.1V + Qwen2-VL ViT; decoupled-head decode fix)
qwen3-tts-0.6b TTS (codec) ADMITTED (structural) ➖ decoder-only; talker deferred
bk-sdm-tiny-onnx image-gen ADMITTED (visible) ➖ browser-fleet viewer renders ONNX; mesh doesn't front it
flux-1-schnell image-gen ADMITTED (visible) ✅ VAE-decode Flux VAE-decode proven via src/bin/flux-image-server.rs + src/model_ltx_vae.rs; MMDiT cos≈1.0
flux-dev image-gen ADMITTED (structural) 🔲 header-parse only; real gen = net-new kernels; gateway 404s
ltx-video video-gen ADMITTED (visible) 🔲 gpu/≥24GB shares model_ltx_vae.rs VAE; full gen GPU- / ≥24 GB-RAM-gated (cuda_matmul.rs ~4–5B/24 GB L4 budget)
gemma-4-31b-it vision/text ADMITTED 🔲 gpu/≥24GB ~17 GB Q4_K + 4 GB fp32 lm_head near the 24 GB L4 ceiling — GPU- / big-RAM-gated

C′. Resumable checkpoint plane + SSM hosting (fat-station / VM, out of band from the CF mesh)

A second model surface runs as native Rust fat-stations / VMs, distinct from the Cloudflare warm mesh (SOVEREIGN-MESH-ZOO.md). It adds a checkpointable, resume-anywhere plane so a forward can be paused and resumed on a fresh host (see the resumable-inference vCPU arc).

Checkpoint endpoints

  • /kv-session — attention KV capture/resume for the dense decoders: Llama + Gemma (open-source/gnosis/distributed-inference/src/bin/fat-station.rs).
  • /ssm-state — recurrent state capture/resume for all SSM models + qwen35moe (open-source/vcpu-infer-ssm/src/bin/ssm-host.rs).
  • Phi-3 deferred — kernel wired but its forward_range_attn_chunk returns the ENCODER_GAP error (no .knot yet / known forward bug) — distributed-inference/src/model_phi3.rs.

SSM / hybrid / recurrent family (open-source/vcpu-infer-ssm/src/{lib.rs,math.rs,model.rs}):

Model family checkpoint host Note
rwkv7-2.9b-world RWKV-7 /ssm-state ✅ ssm-host trie tokenizer (own family); O(1) state
mamba-2.8b Mamba-1 S6 /ssm-state ✅ ssm-host gpt-neox 50k vocab tokenizer step
falcon-mamba-7b Mamba-2 /ssm-state ✅ ssm-host fits an L4; Falcon ~65k vocab
jamba / falcon-h1 hybrid (attn+SSM) /ssm-state ✅ ssm-host hybrid layers (lib.rs)
qwen35moe-heretic hybrid Mamba + MoE /ssm-state ✅ fat-station Qwen35MoePipeline (model_qwen35moe.rs); knot rebuild owed

D. Candidates / prey (backlog — NOT admitted)

flux2-klein-4b · qwen-image · qwen-image-edit · z-image-turbo · sana-sprint/sana · pixart-sigma/delta · auraflow-v0.3 · kandinsky-2.2 · sdxl/sd1.5 ecosystem (non-Apache, research-only). → All need an artifact + per-component admission gate before any lane work; tracked in §"Candidate" of ADMITTED_MODELS.md.

Reverification status roll-up

  • ✅ live-confirmed (5): qwen2.5-0.5b, tinyllama-1.1b, deepseek-r1-1.5b, mistral-7b, llama-3.2-3b.
  • ⏳ admitted + config-correct, mesh probe pending (5): qwen3-4b, qwen2.5-7b, qwen3-8b, loki-8b, gemma4-12b.
  • ❌ admitted but broken on lane (1): gemma3-4b-it (loader SIGBUS — engine work).
  • 🔲 not wired to a live lane (text large/SSM): gemma4-31b(+heretic), llama-3.3-70b, qwen2.5-72b, mamba-2.8b, falcon-mamba-7b, rwkv7-2.9b, jamba-1.5-mini, qwen35moe-heretic, gemma-4-E4B.
  • ➖ multimodal admitted on their own lanes (not the text grind).

Working method (per model, bottom-up)

  1. Trigger the swap (warm); confirm [openai-server] tokenizer=<expected> + [Pipeline] knot config … rope_theta=… in logs.
  2. Real request on the serving lane; compare argmax/first-token to the qspec expect (e.g. Paris 12366).
  3. PASS → record grade (tok/s). FAIL → root-cause: tokenizer family · chat template · rope/arch · loader (SIGBUS) · deployed digest — before moving up.
  4. Batch registry fixes into ONE mesh rebuild; deploy by the build's output digest, never the floating mesh-v1 tag.
  5. Admission (offline qspec) and live-serving are DIFFERENT gates — a model can be ADMITTED yet fail on its lane. Reverify both.