Distributed Inference — Architecture

This document is the consolidated reference for the cobordism + layer-station architecture that serves qwen-coder-7b across Cloudflare Workers + Durable Objects. It aggregates the inline design notes that live alongside the code. For the canonical text on any given subsystem, follow the file:line pointers into the source — the inline comments are SSOT, this doc indexes them.

Readers in a hurry: §1 (topology), §5 (knot format with the encoder warning), and §8 (known traps) are the highest-value sections.

1. Cluster topology

The serving cluster for qwen-coder-7b is a fixed product of two pinned dimensions:

• 2 layer stations — each pins a contiguous transformer-layer range. Entry station owns layers [0, 14); exit station owns [14, 28). Source: apps/worker-node1/src/index.ts top docblock + parseSpecialization (lines 1–32, 184–225).
• 8 cobordism shards — each pins a disjoint slice of the embedding vocabulary. Source: apps/worker-cobordism/README.md lines 1–32 + the fan-out block in open-source/gnosis/distributed-inference-host/src/pipeline.ts lines 238–279.

Vocabulary partition

The qwen-coder-7b tokenizer reports vocab_size = 151,936 (the encoder pads to 152,064 inside the knot — see notes in apps/worker-cobordism/src/index.ts lines 287–292 about why we use the qtype byte formula instead of dividing embedMeta.length / vocabSize).

The math the cluster relies on:

vocab_size = num_shards × shard_size
151,936    = 8           × 18,992

Each shard owns [VOCAB_START, VOCAB_END) with VOCAB_END - VOCAB_START = 18,992, the ranges meet end-to-end with no gaps and no overlaps. Out-of-range tokens at any shard are emitted as the all-zero embedding vector. See apps/worker-cobordism/src/index.ts lines 78–100 for the parser, and lines 360–369 for the out-of-range zero-fill that makes the Frobenius merge in §3 work.

Layer split

Two stations, both 14 layers wide:

station 0 (entry):  start_layer=0,  end_layer=14, role="entry"
station 1 (exit):   start_layer=14, end_layer=28, role="exit"

SPECIALIZATION_* env vars in each station's aeon.toml carry these. The boot path keys its module-level bootedStation cache by ${KNOT_KEY}|${role}|${start}-${end} so a config change inside the same isolate triggers a re-boot (apps/worker-node1/src/index.ts lines 266–276).

Roles drive resident-tensor selection: entry loads token_embd_weight (today only used as fallback — the cobordism mesh handles embeddings), exit loads output_norm_weight + output_weight (with a tied-embedding fallback when the knot omits output_weight). See lines 300–363.

Why these two pinnings together

Layer-pinning solves the per-replica weight footprint for the matmul stack; vocab-pinning solves the per-replica weight footprint for the embedding table. Qwen-coder-7b's token_embd_weight alone is ~306 MB at Q4_K (152,064 × 3,584); the full tensor doesn't fit in a Worker's 128 MB request budget. Splitting eight ways drops it to ~38 MB per shard, and we range-fetch one row at a time from R2 per request anyway (§2), so the resident footprint is tiny.

2. Fluid memory invariant

Statement. No layer-station Worker holds more than the bytes for a single phase of a single layer at any point during forward inference. Tensors are loaded immediately before the WASM call that consumes them and evicted immediately after.

The pattern lives in apps/worker-node1/src/index.ts handleForward, lines 615–691:

for (let l = station.spec.startLayer; l < station.spec.endLayer; l++) {
  const attnNames = await station.loadLayerPhase(l, 'attn');
  station.pipeline.forward_range_attn_chunk(...);
  station.backend.evict(attnNames);

  const gateNames = await station.loadLayerPhase(l, 'ffn_gate');
  station.pipeline.forward_range_ffn_gate_chunk(...);
  station.backend.evict(gateNames);

  const upNames = await station.loadLayerPhase(l, 'ffn_up');
  station.pipeline.forward_range_ffn_up_chunk(...);
  station.backend.evict(upNames);

  const downNames = await station.loadLayerPhase(l, 'ffn_down');
  station.pipeline.forward_range_ffn_down_chunk(...);
  station.backend.evict(downNames);
}

Four phases per layer × 14 layers = 56 load/evict cycles per station per forward. Each cycle holds at most one phase's worth of tensors in WASM linear memory.

Why FFN gate and up are split

gate_proj and up_proj for qwen-coder-7b are roughly 50 MB each at Q4_K. A combined forward_range_ffn_gate_up_chunk call (which still exists for callers that don't care about memory — see wasm_bindings.rs lines 337–349) holds both resident at once for the duration of the swiglu, and that's ~100 MB just for the FFN tensor pair, plus everything else the pipeline is holding. Splitting into separate forward_range_ffn_gate_chunk / forward_range_ffn_up_chunk lets the host evict gate before loading up.

The Rust split lives in:

• open-source/gnosis/distributed-inference/src/wasm_bindings.rs lines 309–335 (gate-only and up-only entry points, with the rationale comment at 309–312).
• apps/worker-cobordism/src/dequantize.ts lines 1–75 (broader fluid-memory motivation: why we range-fetch rows instead of preloading tensors).

Boot resident set

Boot does load some tensors eagerly — only the small ones the pipeline needs across every forward (output_norm_weight, output_weight on exit, formerly token_embd_weight on entry but no longer required because the cobordism mesh handles embeddings). See apps/worker-node1/src/index.ts lines 300–369. The big per-layer matmuls are explicitly not preloaded (apps/worker-node1/src/index.ts lines 365–369).

128 MB worker budget

The total resident weight in a Station Worker, mid-forward, is bounded by:

resident_floor + max_phase(layer L) + activations(prompt × hidden × 4)

For qwen-coder-7b Q4_K_M with our chunkSize of floor(P/2): resident floor is ~6 MB (norms + small tensors), max phase is ~50 MB (ffn_up), activations for a 512-token prompt at hiddenDim 3584 are ~7 MB. Total ~63 MB, well under the 128 MB ceiling. The split-FFN design is what keeps us off the ceiling.

3. TQFT cobordism merge math

Frobenius identity

The Atiyah-Segal TQFT cobordism functor models the embedding lookup as a multiplication μ : A ⊗ A → A, where A is the residual-stream vector space and ⊗ is the disjoint vocabulary partition. In practice the multiplication is elementwise addition of disjoint partials.

The diagram (apps/worker-cobordism/README.md lines 25–32):

shard 0 →  [ e0 e1 0  0  0  0  0  0 ]
shard 1 →  [ 0  0  0  e1 0  0  0  0 ]
shard 2 →  [ 0  0  e2 0  0  0  0  0 ]
                  …
              ↓ μ (elementwise add)
              [ e0 e1 e2 e3 e4 e5 e6 e7 ]

The reason this works without aliasing is the disjoint-vocab construction: for any token t in the input, exactly one shard satisfies t ∈ [VOCAB_START, VOCAB_END). The other seven shards emit zero vectors for that token's slot. Summing eight Float32Arrays produces the correct embedding row in the output position with no double-counting and no missing tokens.

Coordinator merge (canonical comments)

Source: open-source/gnosis/distributed-inference-host/src/pipeline.ts lines 238–279. The block heading "TQFT Cobordism" carries the canonical prose; the loop reduces over shards:

const partialResults = await Promise.all(
  this.cobordismStations.map((cs) =>
    cs.station.cobordismEmbed({ tokens, startPos, vocabStart, vocabEnd, ... })
  ),
);
allEmbeddings = new Float32Array(numTokens * hiddenDim);
for (const partial of partialResults) {
  for (let j = 0; j < totalFloats; j++) {
    allEmbeddings[j]! += partial[j]!;
  }
}

This is currently Promise.all. Section 8 logs a follow-up to convert this to Promise.allSettled so a single shard's failure does not poison the entire prefill. In principle the merge stays correct under partial shard failure: if shard k fails, you still get correct embeddings for every token not in shard k's range. Tokens in k's range come back as zero, which the model will see as the all-zero embedding row — wrong, but localized rather than fatal.

Per-shard zero behavior

The shard's "I don't own this token" signal is the all-zero vector, set by leaving the output buffer untouched at the right offset:

// apps/worker-cobordism/src/index.ts:360-369
if (tokenId < vocabStart || tokenId >= vocabEnd) {
  // Out of range for this shard — leave the output slot as zero.
  continue;
}

The output buffer is initialized via new Float32Array(tokens.length * hiddenDim) (line 352), which zero-fills by spec. The merge just relies on that.

4. Cache hierarchy

There are two cache hierarchies in the system: one for embedding rows (per-cobordism-shard) and one for layer tensors (per-layer-station).

Embedding row hierarchy (cobordism side)

L1: CobordismWardenDO RAM           — sub-ms within colo, ~256 MB budget
L2: Workers KV (env.EMBED_CACHE)    — durable across DO restart, 24h TTL
L3: R2 (env.KNOT_BUCKET range read) — cold path

L1 caches raw quantized row bytes (~2 KB per Q4_K row); L2 caches dequantized F32 rows (~14 KB per row at hiddenDim 3584). The size asymmetry is intentional: a 256 MB DO budget holds ~128k Q4_K rows but only ~18k F32 rows. RAM is precious; per-call dequant is cheap.

Source: apps/worker-cobordism/src/cobordism-warden-do.ts lines 1–60, specifically the "Why store raw quant bytes" rationale at lines 21–28.

Each cobordism worker has its own DO instance keyed by:

{knotKey}|{vocabStart}-{vocabEnd}

— see apps/worker-cobordism/src/warden-client.ts wardenIdName(...) and the call site at apps/worker-cobordism/src/index.ts lines 384–388. All isolates serving the same shard hash to the same DO globally, so the shard's hot rows are warm-cached once across the planet, not once per isolate.

Layer tensor hierarchy (station side)

L1: LayerWardenDO RAM           — 1 GiB default budget, LRU + resident heuristic
L2: (reserved — KV not yet wired for layer tensors)
L3: R2 (env.KNOT_BUCKET)        — cold path

Source: apps/layer-warden/src/layer-warden-do.ts lines 1–32 (top docblock) + apps/layer-warden/README.md lines 1–73.

Each layer station has its own warden DO instance keyed by:

{knotId}|{layerStart}-{layerEnd}

Two stations × one warden each = two DO instances total for qwen-coder-7b. Identity is set via POST /configure or X-Knot-Id / X-Layer-Start / X-Layer-End headers on first call (apps/layer-warden/src/layer-warden-do.ts lines 248–253, 469–541).

LRU residence heuristic (layer side)

Mirrors open-source/gnosis/distributed-inference/src/loader.rs::pick_resident_names:

• tensors with norm in the name stay resident
• token_embd_weight, output_norm_weight, output_weight stay resident
• tensors smaller than 1 MiB stay resident
• everything else (the big matmuls) evicts on LRU when over budget

See apps/layer-warden/src/layer-warden-do.ts lines 375–414.

Cobordism warden eviction

Pure LRU bound by MEMORY_BUDGET_BYTES (default 256 MiB). No "always resident" carve-outs — every embedding row is the same size and the same shape, so a uniform LRU is appropriate. See cobordism-warden-do.ts lines 344–354.

When DO RAM is lost

DOs are torn down by the runtime on idle, cold-restart, or deploy. KV survives those events at the cost of slower reads. On DO miss the worker reads R2 and writes both DO RAM and KV; on KV hit it skips R2 but re-warms DO RAM. The current cobordism warden goes straight to R2 on miss without consulting KV — see the note at cobordism-warden-do.ts lines 30–43 ("KV-as-fallback for DO is wired but currently not consulted").

5. Knot file format

The .knot is the on-disk container for a model's quantized weights and config metadata. It is the only thing R2 stores; everything else (metadata indices, tensor offsets) is parsed from the file on first read.

Canonical layout

Source: open-source/gnosis/distributed-inference-host/src/knot-header.ts lines 1–50, plus the encoder at open-source/bitwise/scripts/encode-llama-knot.ts buildKnotBuffer (lines 283–359).

bytes 0..3                  KNOT magic ("KNOT" ASCII)
bytes 4..5                  u16 version (little-endian)
bytes 6..9                  u32 metaLen (UNPADDED JSON byte length, LE)
bytes 10..10+metaLen        JSON metadata, no trailing padding
bytes 10+metaLen..align-1   NUL pad to 4-byte boundary (0..3 bytes of 0x00)
byte  align                 FP64 codec marker (0x40)
bytes align+1..             tensor payload

Where align = (10 + metaLen + 3) & ~3 and the canonical reader's formula computes payloadOffset = align + 1.

The reader: parseKnotHeaderFromBytes (lines 55–108) and readKnotHeader (lines 114–146) handle the two-fetch case where the first probe is too small for the declared metadata.

Encoder/reader contract

The contract that ties the two together:

Where paddingNeeded = (4 - (10 + metaLen) % 4) % 4.

The off-by-one bug we just fixed (2026-04-25)

The previous encoder pre-padded the JSON with trailing spaces and wrote the padded length into metaLen. That produced this on-disk layout:

[magic, version, paddedMetaLen, paddedJSON, FP64, payload]

with FP64 at byte 10 + paddedMetaLen and the payload at byte 11 + paddedMetaLen. The canonical reader's formula aligned(10 + metaLen) + 1 then computed aligned(10 + paddedMetaLen) + 1, which lands one byte past the actual payload start (because for any padded length the encoder produced, 10 + paddedMetaLen is 4M - 1 and aligned(4M - 1) = 4M, so the reader returned 4M + 1 while the actual payload was at 4M).

That off-by-one shifted every Q4_K block's d/dmin super-scale read into the previous block's qs region, producing garbage f16 values (typically ~1e6 magnitude with NaN sprinkled through). Validation against a real qwen-coder-7b knot confirmed the symptom; the fix is to align encoder and reader on a single layout — the canonical one above. The reader was untouched; the encoder now pads after the JSON and writes the unpadded length.

Source: open-source/bitwise/scripts/encode-llama-knot.ts lines 301–333 (full annotated history), and buildKnotBuffer lines 334–359 (the fixed encoder).

WARNING. Any knot encoded before 2026-04-25's encoder fix is layout-broken; re-encode via the cloudbuild pipeline before serving. Reading a pre-fix knot through the canonical reader silently produces NaN-laced logits — the failure mode does not throw, it returns gibberish. Look for payloadOffset values that disagree with the file size minus the tensor payload total to detect a stale knot.

6. Quantization formats supported

GGUF-style block quantization mirrored from llama.cpp/ggml-quants.c. Source: apps/worker-cobordism/src/dequantize.ts lines 1–75 (top docblock), implementations at lines 191–438.

256-element block alignment requirement

Both Q4_K and Q6_K use QK_K = 256 elements per block (matches llama.cpp upstream). Encoders store blocks contiguously across the whole tensor with no per-row padding, so a row of hiddenDim elements consumes exactly (hiddenDim / 256) × bytes_per_block bytes — provided hiddenDim is a clean multiple of 256.

Off-grid hidden dims would slice mid-block and produce garbage. The guard assertBlockAligned enforces this at row-fetch time (dequantize.ts lines 140–152). Both tinyllama (2048) and qwen-coder-7b (3584) are clean multiples of 256.

JS dequant vs Rust dequant

Two implementations exist for historical reasons:

• JS dequant (apps/worker-cobordism/src/dequantize.ts) is used by the cobordism worker because it range-fetches one row at a time. Loading the full embedding tensor into the WASM backend is not an option (306 MB > 128 MB budget). The motivation is documented at dequantize.ts lines 1–35.
• Rust dequant (open-source/gnosis/distributed-inference/src/) runs inside the WASM pipeline for layer tensors that are loaded whole into linear memory by the host shim. The Rust path uses the same block layouts but in compiled form for the matmul-adjacent hot loops.

A future revision could expose a Rust-side embed_range_sliced that takes a per-shard byte slice and a local-index mapping, unifying the two paths. Not yet shipped — JS dequant is the current bridge.

7. Endpoint matrix

What each Worker / DO actually exposes.

Layer station (apps/worker-node1/)

Source: apps/worker-node1/src/index.ts lines 7–32 (canonical wire contract docblock) + fetch handler lines 567–588.

The wire envelope (request id, position, sequence length, forward mode) is decoded by decodeEnvelope in apps/worker-node1/src/wire-frame.ts.

Cobordism shard (apps/worker-cobordism/)

Source: apps/worker-cobordism/src/index.ts lines 27–32 (top docblock) + fetch handler lines 425–440. The 501 stub is at lines 515–527.

LayerWarden DO (apps/layer-warden/)

Routed through a service binding from each layer station Worker.

Source: apps/layer-warden/src/layer-warden-do.ts lines 1–32 (top docblock) + fetch handler lines 418–471. The /invalidate rationale is at lines 557–582.

CobordismWarden DO (apps/worker-cobordism/src/cobordism-warden-do.ts)

Routed through a Durable Object namespace binding from each cobordism shard Worker.

Source: apps/worker-cobordism/src/cobordism-warden-do.ts lines 45–58 (endpoint summary) + fetch handler lines 132–150. The /tensors batch handler with the rationale for batching is at lines 191–295.

8. Recently-found-and-fixed bugs (known traps)

A reference list of failure modes the next engineer should not rediscover the hard way. Each entry points to where the fix lives so you can see the shape of the trap without re-reading the post-mortem.

dequantizeRow F32-only fallthrough

The previous version of dequantizeRow only implemented F32 and silently returned new Float32Array(hiddenDim) (zero vector) for anything else, with a console.warn. Combined with the field-name bug below, every shard in production was producing zero embeddings for every qtype, including F32. The downstream pipeline folded those zero rows through the layer stations and emitted a near-uniform softmax that decoded as repetitive gibberish. Fixed by implementing F16 / Q4_K / Q6_K dequant in apps/worker-cobordism/src/dequantize.ts (rationale at lines 23–34; implementations at lines 191–438).

tensor_type vs type field name

The raw knot JSON metadata uses type (the on-disk encoder field name). The WASM-side parse_tensors_json renames it to tensor_type for layer stations that consume backend-shaped metadata. The cobordism worker reads metadata JSON directly via readKnotHeader and so must read type, not tensor_type — but a previous revision read embedMeta.tensor_type and silently got undefined for every entry, which fell through to the F32-only dequant above. Fixed at apps/worker-cobordism/src/index.ts lines 263–280; see also the lookup fallback in cobordism-warden-do.ts lines 322–336 that tolerates either field name on legacy or current encoder revisions.

kvGet silent cache poisoning

The cobordism KV cache stores dequantized F32 rows. A historical bug in cobordism-warden-do.ts hardcoded F32 row size for Q4_K models, so the DO wrote rows of the wrong byte length into KV. Without size validation on read, those poisoned entries persisted for their full 24h TTL and cascaded silently on every read.

Fixed by validating buf.byteLength === hiddenDim * 4 on KV read and best-effort deleting the key on mismatch — apps/worker-cobordism/src/index.ts lines 130–168. The delete is void-fired so a kv.delete failure never blocks the inference fast path.

Missing ctx.waitUntil on KV writes

Post-response KV writes register through ctx.waitUntil(...) so the runtime keeps the isolate alive long enough to complete them. Without waitUntil registration, the writes were orphaned by isolate teardown and the cache silently failed to persist. Fixed by threading ctx.waitUntil through computePartialEmbeddings — apps/worker-cobordism/src/index.ts lines 322–344 (rationale) and 416–418 (call site). Also see the handleCobordismEmbed invocation lines 487–495.

Promise.all on cobordism merge (now allSettled with degradation)

The Pipeline coordinator currently uses Promise.all to fan out to cobordism shards (pipeline.ts line 247). One shard timeout / fault fails the entire prefill. The mathematics of §3 says we should be able to degrade gracefully — tokens in the failed shard's range come back as zero, but tokens in other shards' ranges are still correct.

Action item: convert to Promise.allSettled and log per-shard failures without poisoning the merge. See the inline note at the top of tensors in cobordism-warden-do.ts lines 246–252 explaining why row-level partial failure within a single shard is not a recovery scenario (the whole shard goes red), but shard-level partial failure across the merge is.

Encoder/reader payload-offset off-by-one

The big one. See §5 above for the full story. Pre-fix encoder padded JSON inside the metadata length; canonical reader assumed unpadded length. Result was a one-byte payload offset shift that turned every Q4_K super-scale read into garbage. Fixed at open-source/bitwise/scripts/encode-llama-knot.ts buildKnotBuffer (lines 334–359). Re-encode any pre-2026-04-25 knot.

loadLayerPhase silent skip on missing tensor

The previous loadLayerPhase silently dropped missing tensors and called fetchAllTensors with a partial list, leaving the WASM backend with stale or uninitialized memory at the missing slots. The forward step then produced garbage residuals with no error signal.

Fixed by throwing on the first missing tensor with a message that identifies the phase, layer, and tensor name — surface the failure at the first phase that hits the gap, not 28 layers later in scrambled logits. See apps/worker-node1/src/index.ts lines 398–423.

Layer-warden no-invalidate poisoning

When the layer-warden DO returned bytes that the station couldn't decode (malformed AWFR frame, wrong byteLength), the station fell back to R2 and continued — but the same poisoned cache entry would be served on the next request, and the next, forever. Fixed by adding a /invalidate/<name> endpoint to the warden and having stations fire-and-forget invalidate the entry whenever they reject a warden response. See apps/layer-warden/src/layer-warden-do.ts lines 557–582 (DO side) and apps/worker-node1/src/layer-warden-source.ts lines 196–248 (station side).

Layer-warden bucket binding mismatch (was tinyllama, now qwen)

The layer-warden's aeon.toml originally hardcoded KNOT_KEY = "tinyllama-1.1b-q4k.knot" from when the warden was first brought up against the tinyllama smoke test. When we wired qwen-coder-7b through the same warden, the bucket binding still pointed at the tinyllama default and the warden's R2 fallback fetched the wrong file on identity mismatch. Fixed by pushing KNOT_KEY and KNOT_BUCKET config explicitly per-environment in apps/layer-warden/aeon.toml and having the configure call carry an explicit knot_key rather than relying on the env default.

DO scaffolding vs hot-path (cobordism-warden was scaffolding; now active)

The CobordismWardenDO existed for several weeks as scaffolding before the cobordism worker actually routed through it. Until the WardenClient.fetchTensors call landed (the /tensors batch endpoint at cobordism-warden-do.ts lines 191–295), the worker went straight to R2 on every request and the DO's rowCache was always empty. The status note at the top of the DO (lines 12–18, "Architectural status (2026-04-25)") records the transition. If /tensors ever reverts to unused, expect a step-function increase in R2 read costs and a proportional drop in cobordism-shard p50 latency on cache-warm prompts.

Pointers (one-line references)

• Cluster wiring: open-source/gnosis/distributed-inference-host/src/cluster.ts
• Pipelined prefill (Wallington Rotation): open-source/gnosis/distributed-inference-host/src/pipeline.ts
• Knot header parser: open-source/gnosis/distributed-inference-host/src/knot-header.ts
• Knot encoder: open-source/bitwise/scripts/encode-llama-knot.ts
• Layer station Worker: apps/worker-node1/src/index.ts
• Cobordism shard Worker: apps/worker-cobordism/src/index.ts
• LayerWarden DO: apps/layer-warden/src/layer-warden-do.ts
• CobordismWarden DO: apps/worker-cobordism/src/cobordism-warden-do.ts
• JS row dequant: apps/worker-cobordism/src/dequantize.ts
• Rust WASM bindings: open-source/gnosis/distributed-inference/src/wasm_bindings.rs
• Station→warden source: apps/worker-node1/src/layer-warden-source.ts
• Cobordism wire codec: apps/worker-cobordism/src/cobordism-wire.ts
• Cobordism warden client: apps/worker-cobordism/src/warden-client.ts