forgo.cloud
Sign in
Repo workspace

forkjoin-ai/gnosis

Liquid-Memory KV-Save Failure — Root Cause & Fix

distributed-inference-host/LIQUID_MEMORY_KV_SAVE.md
forkjoin-ai/gnosis
Repo overview Open in workbench

Liquid-Memory KV-Save Failure — Root Cause & Fix

Status: root-caused + host-side fix landed. Real perf restoration needs a fat-station-cuda rebuild + moonshine GPU image redeploy (see "Deploy", below).

Symptom (live)

Probe the live mistral service:

curl -s https://edge.affectively.ai/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"mistral-7b","messages":[{"role":"user","content":"Write a paragraph about the ocean."}],"max_tokens":100}' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["_gnosis"])'

With liquid memory ENABLED the _gnosis block showed:

  • liquid_kv_saved: false
  • liquid_miss_reason: "station kv session save failed"
  • liquid_memory: "capture_failed"
  • generation ~0.58 tok/s (62 tok / 106 s).

With liquid memory DISABLED (THOTH_LIQUID_MEMORY=0, the temporary mitigation that is currently set on gnosis-openai-mistral-7b) generation runs at the raw native path ~7 tok/s. So the liquid layer is the cause, not the model.

Architecture (who calls what)

Each model service is one Cloud Run container that runs two processes (open-source/gnosis/moonshine/docker-entrypoint.sh, openai-server arm):

  1. The fat-station — a native Rust GGUF inference server on http://127.0.0.1:8000 (the "single-fat-station" upstream). It owns the model weights and the KV cache.
  2. The openai-server (TypeScript host, distributed-inference-host/src/bin/openai-server.ts) — the OpenAI-compatible front door. It talks to the fat-station over HTTP.

Two decode paths exist on the host:

  • Native fast path (Pipeline.generate → station.generateTokens → fat-station POST /generate). Prefill + decode happen entirely inside the fat-station; KV stays in-process. This is the ~7 tok/s baseline.
  • Liquid-memory path (LiquidMemorySessionManager.generate → Pipeline.preparePrefill + generateFromPrefill). The host orchestrates prefill via POST /embed + chunked POST /forward, then decodes token-by-token via POST /decode-next, and reuses prefixes across requests by saving/restoring the station KV slab via POST /kv-session/save|load|delete.

The liquid path's whole value is cross-request prefix KV reuse (and, within a request, the one-HTTP-call-per-token /decode-next fast decode). When KV-session save fails, BOTH benefits are lost: no reuse, and the decode falls back to chooseTokenThroughStations which re-embeds + re-forwards + pulls full logits over HTTP every token — that is the 0.58 tok/s.

The save wire

  • Host Station.saveKvSession (distributed-inference-host/src/station.ts) does POST {FAT_STATION_URL}/kv-session/save with X-Session-Id + X-Prefix-Length headers. It returns false on HTTP 404 and throws on other non-2xx.
  • LiquidMemorySessionManager.generate wraps that call in .catch(() => false) and, when false, emits missReason: "station kv session save failed" and status: "capture_failed" (liquid-memory-session.ts).

Root cause

The deployed GPU image runs a STALE fat-station-cuda binary that does not implement the /kv-session/* (and /decode-next) routes, so every /kv-session/save returns 404 → host reports capture_failed.

Evidence:

  1. Deployed image is moonshine:baked-mistral-7b-v2-gpu (Artifact Registry, created 2026-06-12). Its env has FLUX_GPU_* flags + the Cloud Run NVIDIA driver, so docker-entrypoint.sh:select_fat_station_bin selects fat-station-cuda (the cuBLAS L4 backend), not fat-station.
  2. fat-station-cuda has no [[bin]] target in distributed-inference/Cargo.toml and is not built by any source-tracked Dockerfile/cloudbuild — it was produced ad-hoc and baked into the GPU image. Its source provenance predates the KV-session routes.
  3. The /kv-session/save|load|delete routes exist ONLY in distributed-inference/src/bin/fat-station.rs and gemma-readout-fix/fat-station.rs. The transport variants (fat-station-uds*, -flow*, -memo, -echo) do NOT have them. A binary built before these routes were added 404s on them.
  4. The route handlers themselves are correct (verified): handle_kv_session_save reads the prefix KV with get_kv_slab and returns 200; host-orchestrated /forward chunked prefill genuinely populates the CPU kv_cache.keys/.values that get_kv_slab reads (the GPU only accelerates the GEMM and copies results back to CPU — KV is never GPU-resident, no cuda cfg-gates touch the KV cache). So a binary built from current source serves KV-session save at 200 and liquid memory works. The bug is purely a stale deployed binary.

Fix

1. Host: detect KV-session capability and never silently degrade (landed)

distributed-inference-host/src/bin/openai-server.ts:

  • probeFatStation() now also returns kvSession. New probeKvSessionSupport() trusts a /health kv_session:false flag, otherwise actively probes POST /kv-session/save (prefix_len=0) at boot: 404 ⇒ unsupported, anything else ⇒ supported (and it cleans up the probe session).
  • liquidMemoryEnabled = LIQUID_MEMORY_ENABLED && !altArch && stationKvSession. When the station lacks KV sessions, liquid memory is forced OFF and the service serves the fast native /generate path (~7 tok/s) instead of the broken 0.58 tok/s liquid path. This is self-healing: redeploying a KV-capable binary re-enables liquid automatically (capability is probed live, not pinned by env).
  • The /health server-info now reports the EFFECTIVE state plus inputs: liquid_memory (effective), liquid_memory_env, station_kv_session.

2. Station: advertise the capability in /health (landed)

distributed-inference/src/bin/fat-station.rs and gemma-readout-fix/fat-station.rs handle_health now emit "kv_session": true and "decode_next": true. Any binary that serves /health from this handler also has the routes, so the host's /health fast-path is truthful and avoids the active probe on capable stations.

3. The actual perf restoration: rebuild fat-station-cuda from current source

The host fix stops the slow-path degradation but, on the current stale GPU image, liquid memory will be auto-disabled (back to 7 tok/s, not the slow 0.58). To get liquid memory ON and FAST (KV/prefix reuse, expected to BEAT 7 tok/s), the fat-station-cuda binary baked into the moonshine GPU image must be rebuilt from current source (which has the /kv-session/* + /decode-next routes) with --features cuda, and the image redeployed.

How to verify

After a redeploy with a KV-capable fat-station-cuda and THOTH_LIQUID_MEMORY unset/1:

  1. Server-info:
    curl -s https://edge.affectively.ai/health | python3 -m json.tool | grep -E 'liquid_memory|station_kv_session'
    Expect station_kv_session: true and liquid_memory: true.
  2. Chat probe (same as the symptom probe). Expect in _gnosis: liquid_kv_saved: true, liquid_memory: "miss" on the first call and liquid_memory: "hit" on a repeat of the SAME prompt, and tok/s ≥ the 7 tok/s native baseline (faster on a repeat prefix).
  3. Headers also carry X-Thoth-Liquid-KV-Saved: true and X-Thoth-Liquid-Memory: hit.

If fat-station-cuda is still stale (404 on /kv-session/save), the host now auto-disables liquid (station_kv_session: false, liquid_memory: false) and serves the native 7 tok/s path — no 0.58 tok/s degradation.

Deploy

The mitigations currently set on gnosis-openai-mistral-7b (THOTH_LIQUID_MEMORY=0, MOONSHINE_PREFILL_WINDOWS=0, plus the off-by-default MOONSHINE_SKYMESH_CACHE/GNOSIS_AMPLITUHEDRON_COORDINATOR) should be revisited once the rebuilt image is live.

  1. Rebuild the moonshine GPU image with a current fat-station-cuda (--features cuda) baked in. The source-tracked CPU cloudbuild is open-source/gnosis/gemma-readout-fix/cloudbuild.moonshine-baked-mistral-7b.yaml (builds Dockerfile.moonshine, tag moonshine:baked-mistral7b-tp1). The deployed -v2-gpu image was built ad-hoc and is NOT reproduced by a tracked cloudbuild — the GPU/cuBLAS build recipe must build the fat-station bin with --features cuda as fat-station-cuda and COPY the current distributed-inference-host/src (which carries this host fix). This is a GPU container build; DO NOT trigger it blindly — confirm GPU/build budget first (note: the project's GPU quota is currently saturated; a second concurrent GPU revision cannot be provisioned).
  2. Deploy: push the rebuilt image and update the Cloud Run service to it, then unset the THOTH_LIQUID_MEMORY mitigation:
    gcloud run services update gnosis-openai-mistral-7b \
  --project neutral-418500 --region us-central1 \
  --image <rebuilt-gpu-image> \
  --remove-env-vars THOTH_LIQUID_MEMORY
  3. Verify per "How to verify" above.

Even WITHOUT a rebuild, deploying just the host fix (re-baked into the image's distributed-inference-host/src) is safe: it turns the broken 0.58 tok/s liquid path into the 7 tok/s native path until the cuda binary is refreshed.