forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

Deaths integration contract

distributed-inference/DEATHS_CONTRACT.md
forkjoin-ai/gnosis

Deaths integration contract

The Five Deaths are implemented as standalone Rust modules — one file per death — composed into model.rs and wasm_bindings.rs by a single foreground integration step. Subagents own files; foreground owns linking.

Each death's module must match the surface declared here so:

  • subagents writing the modules can work in parallel without reading each other,
  • the foreground integration is mechanical (mod x; pipeline.x = X::new(); …),
  • a regression that re-empties one module doesn't leak into the others.

Death #1 — src/matvec_memo.rs

pub const OP_Q: u8 = 0;
pub const OP_K: u8 = 1;
pub const OP_V: u8 = 2;
pub const OP_O: u8 = 3;
pub const OP_FFN_GATE: u8 = 4;
pub const OP_FFN_UP: u8 = 5;
pub const OP_FFN_DOWN: u8 = 6;
pub const OP_LM_HEAD: u8 = 7;
pub const OP_LM_HEAD_RANGE: u8 = 8;
pub const OP_LM_HEAD_PARTIAL: u8 = 9;
pub const OP_OUTPUT_PROJ: u8 = 10;

pub struct MatVecMemo;

impl MatVecMemo {
    pub fn new() -> Self;
    pub fn set_enabled(&mut self, on: bool);
    pub fn set_tau_squared(&mut self, tau_sq: f32);
    pub fn set_max_entries(&mut self, n: usize);
    pub fn clear(&mut self);
    pub fn stats(&self) -> (u64, u64, u64);
    pub fn detailed_stats(&self) -> Vec<(u32, u8, u32, u32)>;
    pub fn probe(
        &mut self,
        layer: usize,
        op: u8,
        rows: usize,
        cols: usize,
        qtype: crate::model::QType,
        input: &[f32],
    ) -> Option<&[f32]>;
    pub fn insert(
        &mut self,
        layer: usize,
        op: u8,
        rows: usize,
        cols: usize,
        qtype: crate::model::QType,
        input: &[f32],
        output: Vec<f32>,
    );
}

pub fn cached_mat_vec(
    memo: &mut MatVecMemo,
    out: &mut [f32],
    x: &[f32],
    w: &[u8],
    qtype: crate::model::QType,
    rows: usize,
    cols: usize,
    n_batch: usize,
    out_stride: usize,
    layer: usize,
    op: u8,
    raw_dispatch: fn(&mut [f32], &[f32], &[u8], crate::model::QType, usize, usize, usize, usize),
);

pub fn cached_mat_vec_f32(
    memo: &mut MatVecMemo,
    out: &mut [f32],
    x: &[f32],
    w: &[f32],
    rows: usize,
    cols: usize,
    n_batch: usize,
    out_stride: usize,
    layer: usize,
    op: u8,
    raw_dispatch: fn(&mut [f32], &[f32], &[f32], usize, usize, usize, usize),
);

pub fn cached_mat_vec_f32_with_context(
    memo: &mut MatVecMemo,
    out: &mut [f32],
    x: &[f32],
    w: &[f32],
    rows: usize,
    cols: usize,
    n_batch: usize,
    out_stride: usize,
    layer: usize,
    op: u8,
    context_hash: u64,
    raw_dispatch: fn(&mut [f32], &[f32], &[f32], usize, usize, usize, usize),
);

Foreground wiring: add memo: MatVecMemo to NativeLlamaPipeline, route the Gemma/Qwen chunked and single-token layer matmuls through cached_mat_vec(&mut self.memo, ..., &mat_vec_dispatch_batch), expose memo_enable/memo_set_tau_squared/memo_set_max_entries/memo_clear/memo_stats/memo_detailed_stats in wasm_bindings.rs.

Death #1 hitlist

These are the known matmul surfaces not yet first-class under the memo / resident / durable-cache boundary. Keep this queue ordered by expected Monster impact and only mark an item closed after a focused regression test proves a warm identical input skips or hydrates.

Closed in this pass:

Surface Closure
LM head full/range/partial lm_head_only, lm_head_range, and lm_head_partial now route through cached_mat_vec with distinct op ids and a shape-aware key. model::tests::lm_head_full_range_and_partial_use_matvec_memo proves full/range/partial warm hits and prevents full-vocab/range/partial bucket collision.
Durable matvec output cache seam @a0n/knotgraph/layer-cache now exports R2MatVecOutputCache, an exact-key R2 helper for compact F32 matvec outputs. getOrCompute hydrates from R2, computes on miss, and writes through best-effort so storage failures do not fail inference.
Worker-side durable matvec hydration optimizedTiledMatVec now has an optional durable-cache hook. LayerLiquidDO installs the KnotGraph/R2 cache when EDGE_MATVEC_OUTPUT_R2_CACHE=1, using the R2 prefix as model identity; known-qtype exact hits return before any KV/R2 weight read.
Phi-3 decode/lm-head matvec coverage Phi3Pipeline now owns a resident MatVecMemo; fused QKV slices, attn output, fused gate/up halves, ffn down, and lm_head route through cached_mat_vec while preserving the existing fused tensor slicing.
Native gnosis-uring FFI matvec admission Native Q4K/Q6K/F32 exported matvec kernels now have an exact resident tier keyed by qtype, dimensions, weight pointer, and input fingerprint, with FFI stats/clear/config controls for benchmark attribution.
Whisper decoder F32 projections WhisperDecoder now enables an exact resident F32 memo for cross K/V prep, self/cross projections, MLP projections, and tied lm_head. WASM/NAPI wrappers expose memo enable/stats/clear for benchmark attribution.
Durable matvec output cache rollout default Mesh and all 70B node Worker configs now default EDGE_MATVEC_OUTPUT_R2_CACHE=1 with the shared matvec-output-cache/v1 prefix, so distributed-inference mesh lanes hydrate exact matvec outputs from R2 by default.
TTS acoustic F32 projections cached_mat_vec_f32 admits already-materialized fp32/fp16-dequant weights into the exact resident memo. TtsAcousticModel now routes mel projection plus encoder Q/K/V/O/fc1/fc2 through it and exposes memo enable/stats/clear.
Speech adapter F32 projections model_speech_adapter is now part of the crate module map and routes its Q-Former K/V/Q/O/FFN/final projections through cached_mat_vec_f32 while preserving the existing forward(&self, ...) API via interior memo state. forward_reuses_resident_projection_memo_on_identical_input proves warm exact hits.
Vocoder decoded-weight residency HiFiGANVocoder now keeps a resident decoded f32 tensor cache for immutable conv weights/biases, avoiding repeated backend fetch and fp16/fp32 materialization across forwards. fetch_f32_reuses_resident_decoded_tensor_cache proves warm tensor reuse and explicit cache clear.
Phi-3 attention-score prefix memo cached_mat_vec_f32_with_context adds a caller-supplied discriminator for dynamic F32 "weights"; Phi-3 per-head attention now hashes the live K-prefix and memoizes raw q @ K scores before SWA mask/scale. context_hash_keeps_f32_attention_prefixes_distinct prevents wrong hits when the prefix changes, and existing Phi-3 attention tests pass through the memo-enabled implementation.
Llama/Gemma attention-score prefix memo NativeLlamaPipeline now owns exact per-layer/per-head attention-score memos, enabled by the same memo_enable(true) hotpath switch. All four model.rs attention-score sites hash the live K-prefix and memoize raw q @ K scores before scale/softmax, so Rayon head loops do not contend on the main matvec memo. forward_one_uses_attention_score_prefix_memo proves warm identical decode hits.
Q5 gate/up pair unmasked path The Q5_0 paired gate/up fast path now routes its unmasked branch through two existing cached_mat_vec admissions with distinct OP_FFN_GATE / OP_FFN_UP ids. q5_gate_up_pair_uses_existing_matvec_memo_contract proves the pair can warm-hit without a new compact-output layout.
Single-output masked leakage kernels Q5_0 masked gate/up plus Q4K/Q6K masked down branches now route through cached_masked_mat_vec, which adds a mask-byte context hash and mask-role discriminator to the existing memo key. masked_matvec_memo_key_includes_mask_bytes proves changed masks miss and same masks warm-hit.
Paired output-masked Q5 gate/up The Q5 paired output-masked branch now uses cached_q5_pair_output_masked, which keys both gate/up outputs by the output-block mask and block length before admitting distinct OP_FFN_GATE / OP_FFN_UP compact outputs. q5_pair_output_masked_memo_key_includes_output_mask_and_block_len proves changed output-mask geometry misses and same geometry warm-hits.
Masked memo benchmark attribution bench-matvec now reports raw masked kernels alongside exact warm memo paths. On the synthetic Qwen-shaped release run (--iters 50 --vocab-rows 8192), warm memo measured Q5 masked gate at 1.94 us, Q5 output-masked paired gate/up at 8.34 us, Q4K masked down at 5.14 us, and Q6K masked down at 5.12 us per call, versus raw masked paths in the 0.58-10.34 ms range under local Rayon scheduling noise.
Priority Surface Current state Next bite
P2 Whisper encoder Encoder batch matvecs use direct mat_vec_f32_batch; decoder now has resident F32 memo coverage. Decide whether encoder batch memo pays off versus the cost of hashing every 1500-frame batch row.
P2 Remaining Pneuma conv output surfaces TTS acoustic, speech adapter F32 projections, and vocoder decoded-weight residency are covered. Flow/vocoder output tensors are convolutional, not simple matvec dispatch shapes. Define a separate conv-output cache contract before wiring; do not force them through the matvec key layout.
P3 Monster masked memo shootout Masked FFN leakage kernels are wired and synthetic bench-matvec attribution is in place. Run an end-to-end Monster profile with leakage modes enabled to measure real hit rates and cache residency pressure.

Death #2 — src/amplituhedron.rs

pub struct AmplituhedronCache;

impl AmplituhedronCache {
    pub fn new() -> Self;
    pub fn set_enabled(&mut self, on: bool);
    pub fn set_max_volumes(&mut self, n: usize);
    pub fn clear(&mut self);
    pub fn stats(&self) -> (u64, u64, u64);
    pub fn capture(
        &mut self,
        prefix_hash: u64,
        prefix_len: u32,
        layer_lo: u16,
        layer_hi: u16,
        tail_residual: Vec<f32>,
        kv_slab: Vec<f32>,
    );
    pub fn replay(
        &mut self,
        prefix_hash: u64,
        prefix_len: u32,
        layer_lo: u16,
        layer_hi: u16,
    ) -> Option<(Vec<f32>, Vec<f32>)>;
}

Foreground wiring: add amplituhedron: AmplituhedronCache field to NativeLlamaPipeline. Wasm-bindgen accessors amplituhedron_enable / set_max_volumes / clear / stats / capture / replay. The capture/replay paths know the live KV layout (in model.rs); the cache itself is layout-agnostic — it just stores and returns Vec<f32>.

Death #3 — TS-only, no Rust module

See ./AEON_FLOW_TRANSPORT_PLAN.md. Surface change is in the worker handlers (apps/worker-node1/src/index.ts) and bench coordinator (scripts/bench-trisplit.ts). No wasm_bindings.rs changes.

Death #4 — src/octonion.rs (TBD)

Plan: ./open-source/gnosis/distributed-inference/OCTONION_ROUTING_PLAN.md. Surface to be defined when implementation starts; reserve the file path so the standalone-module discipline holds.

StructuralErrorgle-author rule for model.rs and wasm_bindings.rs

These two files are integration points. Only the foreground session edits them, and only after every standalone module declared above has compiled in isolation. Subagents that need integration-time changes return text patches; the foreground applies them serially.

If a future session finds either file regressed (struct missing, accessor missing), check git diff against the committed baseline and re-apply from the standalone modules. Don't assume in-flight subagents will preserve them.