@a0n/distributed-inference-host

Workspace-shared host shim for the wasm32 build of open-source/gnosis/distributed-inference. Without this package every Worker that runs distributed inference would reinvent: knot header parsing, R2 Range fetching, layer prefetch policy, LRU eviction, and the orchestration sequence between async fetches and synchronous WASM calls.

Getting Started

What: the TypeScript host shim that lets Workers and edge runtimes drive the WASM build of distributed inference.
Why: it centralizes knot parsing, R2 range reads, prefetching, LRU eviction, and WASM call sequencing so every Worker does not invent its own host layer.
How: depend on @a0n/distributed-inference-host from a workspace package and implement a KnotByteSource for your storage backend.
Next: read the install and usage sections, then the src README when adapting the host contract.

Install

This package is a workspace member; consumers depend on it via the workspace protocol:

// in some Worker's package.json
"dependencies": {
  "@a0n/distributed-inference-host": "workspace:*"
}

Usage

import { HostShim, type KnotByteSource } from '@a0n/distributed-inference-host';
import * as wasm from 'distributed-inference-wasm/pkg';

class R2KnotSource implements KnotByteSource {
  constructor(
    private readonly bucket: R2Bucket,
    private readonly key: string
  ) {}
  async rangeGet(start: number, end: number): Promise<Uint8Array> {
    const obj = await this.bucket.get(this.key, {
      range: { offset: start, length: end - start + 1 },
    });
    if (!obj) throw new Error(`R2 miss: ${this.key}`);
    return new Uint8Array(await obj.arrayBuffer());
  }
}

const shim = await HostShim.open(
  new R2KnotSource(env.MODELS, 'gemma3-1b.knot'),
  wasm,
  { lruCapacity: 6, draftMode: false }
);

try {
  const tokens = await shim.generate(new Uint32Array([1, 2, 3]), 32);
  // ... decode tokens
} finally {
  shim.dispose();
}

Carrier Medium

CarrierMediumPipeline is the deterministic core for observer-relative carrier readings. It accepts arbitrary unit-valued cells, projects them through observer bandwidths, and reports each +1 step as evolution or resolution with the fiber/copy-index loss witness needed for future reconstruction work.

import { CarrierMediumPipeline } from '@a0n/distributed-inference-host';

const medium = new CarrierMediumPipeline({
  sourceBandwidth: 54,
  observerBandwidths: [3, 54],
});

const analysis = medium.analyze({
  cells: [
    { id: 'patch-0', unitValue: 2 / 54 },
    { id: 'patch-1', unitValue: 53 / 54 },
  ],
});

console.log(analysis.cells[0]?.readings.map((r) => r.reading));
// ["evolution", "resolution"]

ImageCarrierPipeline is now an adapter over that medium. It turns image patches into carrier cells and preserves patch geometry beside the same fiber witnesses. The north-star use case is deblurring: a later sampler can use the reported fibers to ask which high-band image states are compatible with a low-band collapsed observation.

import { ImageCarrierPipeline } from '@a0n/distributed-inference-host';

const imageCarrier = new ImageCarrierPipeline({
  carrierBandwidth: 54,
  observerBandwidths: [3, 54],
  patchSize: 1,
});

const imageAnalysis = imageCarrier.analyze({
  width: 2,
  height: 1,
  channels: 1,
  sampleRange: 'unit',
  data: new Float32Array([2 / 54, 53 / 54]),
});

console.log(imageAnalysis.patches[0]?.readings.map((r) => r.reading));
// ["evolution", "resolution"]

This is not a learned vision kernel and it does not reconstruct pixels by itself. It produces carrier metadata that image-native stations, deblur samplers, or ViT kernels can consume through the same mesh transport shape.

Light Model Artifacts

light-model-artifact is the shared TypeScript adapter for the reversible Light model bundle exported by apps/brain-backwards-from-light. It validates the ai.forkjoin.light.model-bundle schema, reconstructs retained source-estimate, retinal-field, and cortical-mode RGBA previews, and projects the bundle into a Monster Mesh artifact summary for Monster Studio.

import {
  parseLightMonsterMeshArtifact,
} from '@a0n/distributed-inference-host/light-model-artifact';

const artifact = parseLightMonsterMeshArtifact(await file.text());
console.log(artifact.metrics.retainedEnergy);

The adapter is browser-safe and does not call paid AI, Workers AI, or network inference. It models the retained bundle state; it does not claim that a PNG alone can recover the original photo.

Stateless Mesh Projection

buildStatelessMeshGrid creates the reusable UV/index buffers for raw WebGPU or WebGL2 extrusion. The per-image state stays in depth and alpha texture planes: the vertex shader samples depth for Z displacement, and the fragment shader discards pixels below the alpha threshold.

For a solid extrusion, use solid: true for front/back planes and add buildAlphaBoundaryWallIndices when the alpha mask changes. Fragment discard gives a clean shell silhouette; real side walls need explicit boundary indices.

What the shim handles

Knot header parse: KNOT magic, version, metadata length, padded JSON block, codec marker. Computes the payload offset as ((10 + meta_len + 3) & ~3) + 1.
Resident tensors: token_embd_weight, output_norm_weight, and the optional output_weight are loaded once at open() and never evicted.
Per-layer prefetch: the nine canonical tensor names per transformer block, fetched in parallel and uploaded to the WASM backend.
LRU eviction: oldest-touched layer drops out when capacity is exceeded. Re-touching a resident layer bumps it to MRU.
Spectral governance: spectral-noise-monitor.ts classifies normal, pre-collapse, and boundary-collapse carrier states. Its fingerprint-level Bule pressure mirrors Gnosis.TopologicalMetabolismBuleyBridge.fingerprintBuleyUnit: saturated brown pressure is one corrective contract over the Hexon/two-head ceiling. The same monitor can attach an attention closure lift decision that mirrors Aether's runtime surface: the Aeon 8-head cut is classified as aliased noise, and Dark Deceptacon policy lifts it to the source-resolution separated trace when the residue/aliasing threshold is crossed. The assessment also carries a work policy and saved-work estimator for preserving heads, capping speculation, and deferring compression while the lift is required, plus theorem-lineage metadata for the Lean optimizer-admission boundary.
Mesh carrier JSON (mesh_carrier_assessment/v1): workers can attach a stable assessment blob per hop (spectral + optional attention closure + optional deblur FRF fold). The host exports encodeMeshCarrierAssessmentV1Json, mergeDeblurFoldIntoMeshCarrierJson, and attachDeblurEnsembleToMeshCarrierAssessmentJson so coordinators can fold forkRaceFoldDeblur diagnostics onto an existing serialized assessment without re-running spectral classification.
Terminal prosody pressure: pipeline.ts can observe or bias Reynolds/Phi sentence-wrap pressure near finite decode horizons. The OpenAI shim defaults to bias mode, keeping native /decode-next hot until the terminal break and then using logits biasing to close/stop cleanly. TERMINAL_PROSODY=observe emits telemetry without changing logits or remote decode routing; TERMINAL_PROSODY=off disables the controller.
Local TTS audio: tts-audio.ts backs the OpenAI-compatible POST /v1/audio/speech route with local WAV/PCM synthesis and explicit playback modes. MOONSHINE_TTS_AUDIO_MODE=auto|host|pulse|alsa|file resolves to host relay on macOS, ALSA on Linux when /dev/snd is mounted, and file output otherwise. The route is independent of chat completion streaming.
Speculative prefill windows: prefill-window.ts backs explicit POST /prefill/windows, PATCH|GET|DELETE /prefill/windows/:id, and X-Moonshine-Prefill-Window attach on /v1/chat/completions. Windows are inert unless a caller creates and attaches one, default to one active 512-token window with a 15s TTL, and can be disabled with MOONSHINE_PREFILL_WINDOWS=0. Attach is exact-prefix only; mismatch, expiry, or KV restore failure falls back to normal cold prefill and reports X-Moonshine-Prefill: mismatch|expired|miss|disabled.
Agentic superserver chat: agentic-chat.ts wraps any local text generator with OpenAI-compatible multi-step chat, ForkJoin MCP superserver tool preflight, tool execution, and C0-C3 metacognitive gates. The Moonshine OpenAI server defaults /v1/chat/completions to bare model mode; AGENTIC=1, X-Zedge-Agentic: tools|auto|1|true, or matching request body fields opt into the tool loop. MCP preflight is lazy and request-scoped unless tools are supplied directly.
Thoth prompt feedback contract: thoth-prompt-feedback.ts mirrors native /prompt-feedback JSON and requires the failure-scribe admissibility theorem plus the rejection-residue preservation theorem before callers can type a response as prompt-injectable context.
Fork/race/fold primitive: frf.ts is the generic FORK (concurrent tasks) → RACE (filter survivors) → FOLD (merge) used across the mesh gather path. Its result carries the canonical ternary Verdict (@a0n/bitwise/triton-verdict): Accept (all forks fulfilled), Abstain (partial survivors — recoverable/defer), Decline (all rejected) — restoring the middle the old private sawRejection boolean squashed. Acting on Abstain (ship the partial fold, backfill only the failed slots) avoids the whole-batch retry the bool forced; __tests__/frf-race-speed.bench.test.ts measures the payoff (up to ~22x less fork-work / ~15x less wall-clock at N=16, p=0.20, where binary retry blows up ~`1/(1-p)^N). Early-commit safety is theorem-backed by gnosis-math Gnosis.TritonForkRaceFold`.
Maha virtual sharding: amplituhedron-sharding.ts maps every physical AlphaJump mesh onto 21 stable virtual shards. One station owns all 21, two split 11/10, three split 7/7/7, and larger meshes keep the same virtual coordinate system. The local benchmark in scripts/amplituhedron-maha-sharding-bench.ts shows async fanout speedups of 3.72x at 21 physical stations and 5.94x at 42 stations by avoiding broad replay/capture probes.

Source Map

src: host shim, station transports, pipeline, and agentic chat exports.
scripts: Zodiac attack orchestration helpers including optional Colab bootstrap/runner surfaces.
README-DEBLUR.md: image/video/audio recovery APIs and fork/race/fold deblur ensemble behavior.
README-STYLE-TRANSFER.md: classical style transfer, AdaIN adapter contract, and FRF ensemble style fold APIs.
tests: focused regression tests for host behavior.

Mesh serving & ops docs

MESH_CONCURRENCY_AND_FEDERATION.md: how the mesh serves multiple requests (ack-and-hold) and multiple models (federation peers) on scarce GPUs — plus Q4_K variants, hold-through-reload, the build/cutover runbook, tunables, and the live topology.
MESH_BUILD_DEPLOY_RUNBOOK.md: image build + Cloud Run deploy steps.
MESH_QUALITY_REPORTING_FRAMEWORK.md: decode grade scale (AAA→F) + the Q4_K=Q8 parity record.
MESH_DECODE_PERF_BUDGET.md: per-component decode budget.
MESH_BANDWIDTH_CONSTRAINTS.md: why decode is bandwidth-bound.

Zodiac runtime selection

zodiac-attack now supports an optional Colab lane in addition to local runs.

Local Rust lane (default for workstation/CI):
- python3 open-source/gnosis/distributed-inference-host/scripts/zodiac-colab-run.py --runtime local --engine rust --rounds 8 --output /tmp/zodiac-rust.json
Local GPU lane:
- python3 open-source/gnosis/distributed-inference-host/scripts/zodiac-colab-run.py --runtime local --engine gpu --rounds 8 --iterations 10000 --swarm-size 16384 --output /tmp/zodiac-gpu.json
Colab lane:
- bash open-source/gnosis/distributed-inference-host/scripts/colab-bootstrap.sh /content/emotions
- python3 /content/emotions/open-source/gnosis/distributed-inference-host/scripts/zodiac-colab-run.py --runtime colab --engine auto --mount-drive --drive-output MyDrive/zodiac/zodiac-best.json --rounds 12

Notebook: open-source/gnosis/distributed-inference-host/scripts/Zodiac.ipynb.

What is intentionally out of scope

Streaming per-step inference with prefetch overlap. generate() preloads all layers; replace with explicit prefetchLayer(L) plus a future pipeline.step() API for memory-constrained models.
WASM module loading. Callers pass the wasm-bindgen module in directly so this package stays runtime-neutral (CF Workers, Node, Bun, Deno).

forkjoin-ai/gnosis