fat-station --rknot Runbook

Boot a single fat-station instance from a Resonance Knot (.rknot) with a dense .knot fallback. Single-station validates the path. Multi-station is the same args N times with different --layers windows + a coordinator on top (see § Multi-station Monster Mesh).

Prerequisites

• Release build of fat-station:

  cd open-source/gnosis/distributed-inference
  RUSTFLAGS="-A warnings" cargo build --release --bin fat-station

• A .rknot file (local path; HTTP support not yet wired into RknotBackend::open)
• A dense .knot source (local path OR R2/HTTP URL — KnotWeights::load dispatches)
• R2_PROXY_TOKEN set if the dense source is on the R2 proxy
• KNOT_HTTP_USER_AGENT set to a browser-shaped UA if R2 is fronted by a CF bot filter (the loader.rs::http_range_get path wires both env vars)

Single-station boot — validated 2026-05-04

The exact command we used for the Phase 4.5 boot smoke against Llama-3-70B precise.rknot (k=0.30, bits 8/8/8 from Track B):

R2_PROXY_TOKEN="..." \
KNOT_HTTP_USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
  ./target/release/fat-station \
    --rknot /tmp/llama-70b-precise.rknot \
    --dense "https://r2-proxy.taylorbuley.workers.dev/r2/models/llama-70b.knot" \
    --layers 0..10 \
    --role mid \
    --port 8004 \
    --no-standing-wave

Boot signature (what success looks like):

[fat-station] booting role=mid layers=[0,10) port=8004 weights=/tmp/llama-70b-precise.rknot
[fat-station] spectral path: rknot=/tmp/llama-70b-precise.rknot dense_fallback=https://...
[KnotWeights] HTTP load: https://r2-proxy.taylorbuley.workers.dev/r2/models/llama-70b.knot
[Pipeline] knot config: hidden_dim=8192 layers=80 heads=64 kv_heads=8 head_dim=128 ffn=28672 vocab=128256 rope_theta=500000
[fat-station] pipeline ready in 87656 ms
[fat-station] listening on http://0.0.0.0:8004

pipeline ready lands in ~90s for Llama-70B (lazy block decode in RknotBackend keeps RAM at file size, ~5 GB, not the dense f32 expansion ~97 GB). The dense fallback HTTP load fetches only the header window — actual tensor bytes pull lazily via http_range_get per fetch_tensor call.

CLI flags this adds

Validation is enforced at parse time:

/health on the rknot path

After the fix at model.rs::vocab_size, the /health endpoint reports the canonical dims pulled from the loaded pipeline (not the legacy sniff_dims local-file sniff that fell back to qwen defaults for URL sources). For Llama-70B you'll see vocab_size=128256, hidden_dim=8192 in the health JSON.

Multi-station Monster Mesh

The single-station validation is the building block. To deploy a mesh, spawn N fat-stations with non-overlapping --layers windows all pointed at the same rknot + dense URL, then layer a coordinator on top.

# 8 mid-stations covering 0..80 in 10-layer chunks
for i in 0 1 2 3 4 5 6 7; do
  start=$((i * 10))
  end=$((start + 10))
  port=$((8001 + i))
  R2_PROXY_TOKEN="..." \
  KNOT_HTTP_USER_AGENT="Mozilla/5.0 ..." \
    ./target/release/fat-station \
      --rknot /local/llama-70b-precise.rknot \
      --dense "https://r2-proxy.../llama-70b.knot" \
      --layers ${start}..${end} \
      --role mid \
      --port $port \
      --no-standing-wave &
done

Coordinator: open-source/gnosis/distributed-inference-host/src/station.ts fronts each station as an RPC client. Wire N Station instances at the 8001-8008 endpoints; the coordinator dispatches forward(residual, startPos, seqLen) across them in order.

Cobordism shards (vocab sharding for the lm_head output projection) follow the qwen-coder-7b cluster shape: the 80-layer mesh terminates into K shards each holding vocabSize/K rows of the lm_head matrix. Frobenius merge produces the final logits.

Known limitations

1. RknotBackend::open is local-Path-only. To stream rknot from R2 without the local download, add RknotBackend::open_http(url) analogous to KnotWeights::load_http (the rknot reader uses read_rknot_from_file which would need an HTTP-Range counterpart — the rknot format already supports random per-block access via body_offset/payload_byte_len in the JSON header, so the adapter is just a Range-fetch loop).
2. The dense .knot path supports HTTP today (validated against the 39 GB Llama-70B dense file via R2 proxy). No local download needed.
3. sniff_dims (local-file knot header parser) is now dead code in fat-station — pipeline accessors replaced its single call site. Left in place for now in case a future caller wants it.

What was validated 2026-05-04

The path is structurally sound. Multi-station deploy is the next push and needs cloud infra (or large local disk for N rknot copies).