Resilient in-room R1 mesh — design for shitty devices that brick constantly

Target: throw N (cheap, flaky) Rabbit R1s in a room; they self-organize into a working distributed-inference mesh and keep answering while devices brick, drop WiFi, and rejoin. No node is load-bearing. No central coordinator in the data path.

This builds on the proven pieces: per-node memory sharding (partial loader, cos=1.0), the /embed → /forward(X-Layer-Range) → /lm-head wire across real nodes, and the monster-guard qspec quality gate (the golden star).

Hard assumptions

• Devices brick at random — mid-idle, mid-forward, mid-decode. Constantly.
• WiFi is lossy / partitions. BTLE for discovery is flaky too.
• In one room — low latency, decent bandwidth, but unreliable links.
• A model is sharded into stages (contiguous layer ranges) that each fit a node's ~2.4 GB RAM budget (see R1_MESH_SCALING_PLAN.md).

Principles

1. No load-bearing node. Every stage [a,b) is held by ≥2 replicas. The mesh survives any single brick and degrades gracefully as more die (down to 1 replica/stage = still works, just no headroom).
2. Stateless hops. A forward is a pure function of (residual, start_pos, layer_range). Any replica of a stage can serve any hop → failover = re-send to another replica, no shared state required.
3. Coordinator-free routing. Each node knows the live replicas of the next stage (from gossip) and routes to one; on timeout/error it retries an alternate replica. The caller hits any stage-0 replica; the answer bubbles back.
4. Self-healing membership. Nodes gossip {node_id, model_id, stage, ip:port, health, load, epoch}. Missed heartbeats → evicted from routing. New nodes → assigned an under-replicated stage (RAM-aware) → admitted, then serve.
5. Admission-gated joins. A joining node must pass the monster-guard qspec on its shard before it's added to a stage pool — a corrupt/flaky/lying node can't poison the mesh. (Reuses the golden-star gate per stage.)
6. Fail fast, not hang. If a stage has zero live replicas, the request errors with stage K unavailable immediately rather than hanging — the caller can retry once the mesh re-forms.

KV / decode resilience

The forward uses a per-position KV cache that lives on whichever replica processed the prior tokens. If that replica bricks mid-decode, its KV is gone. Two modes:

• STATELESS (max resilience, default for the experiment): every token re-prefills through a freshly-routed path (no cross-token KV reuse). Brick-proof — any token can be served by any live path. Slower (re-reads the prompt each token) but never wedges.
• STICKY+re-prefill (faster): a sequence pins to a path and reuses KV; on a hop failure the coordinator-node re-prefills the prompt on a new path and resumes. Faster steady state, with a recovery cost on each brick.

Start STATELESS; add STICKY+re-prefill once the membership layer is solid.

Components (build order)

1. /chain with a next-stage replica POOL + failover (MESH FIRST — building now). fat-station --stage a..b --next url1,url2,.... /chain: do my stage, then try the next-stage replicas in randomized order until one returns; bubble the result back. Static pool now; gossip fills it later. This is the resilience primitive — it already survives bricks given a populated pool.
2. Liveness + gossip registry. Periodic heartbeat over WiFi (UDP gossip) + a BTLE beacon advertising {node_id, model, stage, ip}. Each node maintains a live per-stage replica view; evict on missed heartbeats. (Discovery + "find each other".)
3. RAM-aware stage assignment / ring election. On join, a node picks the most under-replicated stage it can fit; ephemeral elector breaks ties; no runtime coordinator.
4. qspec admission per stage. A joiner proves its shard (argmax gate) before serving.
5. Boot service + equip. fat-station as a boot service bound to the WiFi IP; the mass-unknot flashing/setup script provisions binary + shard + manifest at equip time (plug → equip → release).

Failure modes → behavior

In-room experiment (the demo)

N R1s in a room on WiFi. Power them on → they gossip, self-assign replicated stages, admit via qspec, and serve. Run sustained inference (a steady token stream) while you physically unplug/brick random devices — the mesh re-routes around them and keeps answering; plug them back and throughput recovers. The headline: quality-gated distributed inference that survives a room of unreliable $200 devices.

Status

• Proven: sharding, the wire, the qspec gate, correct inference (bowl + qwen3 fixes).

• Component 1 DONE (gnosis 3b0f53ae): /chain server-side peer relay + --next next-stage replica pool + failover (3 s connect-timeout). Verified: entry→exit relay argmax 12095; with a dead replica first in the pool, 3/3 fail over to the live one. Caller hits only the entry node; hops are peer-driven.

• Full stack proven at 7B (2026-05-24): /chain relay over a memory-sharded 7B — mistral-7b split into two 3.6 GB shards (each node loads only its half via the partial loader), caller→entry(shard 0-16)→exit(shard 16-32)+lm-head → 'Paris' @ 13.225. And sharded == monolithic for mistral-7b: residual cos=1.0, same argmax. So peer-relay + per-node memory sharding + correct inference compose end-to-end on a real 7B.

• SOVEREIGN R1↔R1 MESH OVER WIFI — PROVEN (2026-05-24): two R1s on WiFi ("Hawk", 10.0.0.199 + 10.0.0.240). R1#1 (entry, layers 0-12) relays the residual directly to R1#2 (exit, layers 12-24) over WiFi — no Mac/coordinator in the data path; R1#2's log shows the relayed /chain hop. End to end: caller(WiFi) → R1#1(WiFi) → R1#2(WiFi) → argmax 12095 @ 16.82, 766 ms warm, zero USB in the path. Two untethered $200 devices doing quality-correct distributed inference peer-to-peer. The transport-agnostic /chain lit up on WiFi unchanged.

• Component 2 DONE (gnosis 00c8c134 + ac508c5d): UDP gossip membership + liveness (src/mesh_gossip.rs). --gossip-port/--peers/--node-id. Each node advertises its stage + data endpoint; /chain auto-populates its failover pool from LIVE discovered peers (static --next as fallback); /mesh/peers for observability. Direct heartbeats refresh liveness, relayed lists are discovery-only. Verified over localhost: entry auto-discovers 2 exit replicas → /chain routes via the discovered pool (12095, no static next); brick a replica → evicted within the 8 s window → /chain self-heals to the survivor (3/3). Gossip rides house flow_frame::FlowFrames (the aeon wire), VENT-tagged, on a dedicated mesh stream 2010 — distinct from 2002 (the Thoth/pneuma conscious stream) so control traffic never collides with consciousness; one const retargets to 2002.

• Component 2 VERIFIED OVER REAL WIFI (2026-05-24): two R1s on "Hawk" (10.0.0.199 + .240), no static --next. They discover each other via WiFi gossip (2010 FlowFrames) → /chain auto-routes via the discovered pool → 12095; each yields the 2002 brain digest (/mesh/hive: nodes=2, stages [[0,12,1],[12,24,1]], covered 0-24, status "healthy (no redundancy)"). A self-organizing, self-aware untethered mesh. (Restart gotcha: a backgrounded adb -s X shell 'pkill; nohup…' can silently fail to replace the station — verify the new process via its log.)

• Component 5 (equip) DONE (gnosis through 43000911): scripts/r1/equip.sh (per-node) + scripts/r1/mass-equip.sh (whole fleet). Catch-and-release: plug in, run mass-equip, unplug. Each node gets: idempotent binary push; root-aware ops (su 0 sh -c when adb shell isn't root — R1s vary); a kernel wakelock (/sys/power/wake_lock) + stay-awake so a RELEASED node on battery doesn't deep-doze and suspend fat-station; a setsid-detached start that survives USB unplug; a bulletproof restart (root-pkill + wait-until-gone, so a stale station can't hold the port) with a node-id verify (role-only checks miss stale stations); a pushed boot-rejoin hook (host-built then adb push, no nested-quote breakage). mass-equip deals devices round-robin across stages so redundancy grows with the fleet. equip.sh --apk also does the post-flash OS prep itself (launcher, WebView provider, lockscreen off, provisioned), so no external setup script is needed. Verified: one mass-equip.sh -> r1-0 entry + r1-1 exit, both wakelocks held, mesh forms (/chain 12095). Reboot-survival is the in-app node (see R1_INAPP_NODE.md).

  Catch-and-release runbook: flash the device with the mtkclient toolkit in packages/aeon-r1-unknot (the "unknot" flasher) -> scripts/r1/equip.sh --apk (one command: APK + OS prep + self-election conf + launch) -> unplug (keeps serving on WiFi via wakelock+setsid; re-joins on every boot). Inspect: curl <node>:8090/mesh/hive.

• Component 4 (qspec admission) DONE (gnosis through c15a09db): a joining node can't be routed to until it passes the golden-star PARIS gate. Registry gains an admitted flag + opt-in admission_required; an entry/both node with --admit probes each discovered next-stage peer (embed + my layers -> peer /forward + /lm-head -> argmax) and admits ONLY on argmax == --admit-expect (default 12095). next_stage_pool returns only admitted peers; admission survives heartbeats. Verified on localhost: good exit -> ADMITTED, /chain 12095; wrong-expect (bad-node proxy) -> REJECTED, pool empty, /chain 503. --admit threads through equip.sh + mass-equip.sh, so mass-equip.sh --admit gives a qspec-gated fleet. 6 unit tests.

• Component 3 (RAM-aware stage election): DONE + WIRED (gnosis 9be8c515 + the elector): the election is a pure fn recommend_stage_from(peers, model_id, stages, capacity_mb, target_replicas) (the recommend_stage method delegates to it) — counts live replicas per stage, recommends the most under-replicated stage below target_replicas, capacity tie-break. 11 lib tests green; kept off PeerInfo (no ripple into the concurrently-owned fat-station.rs). WIRED at SHARD-ASSIGNMENT time, not fat-station startup: a node's stage is tied to the shard it holds on disk, so a rebooted node must resume its shard — it can't re-elect a stage whose shard it lacks. So the host tool mesh-elect (src/bin/mesh-elect.rs) runs during provisioning: it pulls a live seed's /mesh/peers, reads the joining device's free RAM, and prints the stage to take. mass-equip.sh calls it per device (device 0 seeds as entry; each later device elects vs the live mesh), falling back to round-robin if the seed is unreachable. VERIFIED LIVE: against the running R1 seed (10.0.0.199) a new 2.4 GB node is told 12..24 (the under-replicated stage), exit 0. CAVEAT: the capacity tie-break uses a ~1 GB/layer heuristic tuned for ~7 B shards; for the equal-size 12-layer stages we run today the under-replication signal dominates and the tie-break is moot — refine mb-per-layer when running heterogeneous big-model shards.

• 5b (reboot survival): DONE + PROVEN ON HARDWARE (aeon-shell e18659b; see docs/R1_INAPP_NODE.md). The su boot hook was architecturally dead (this GSI forbids apps from using su or exec'ing /data binaries). The working mechanism: ship fat-station as libfatstation.so in the APK's jniLibs and have the Aeon Shell launcher exec it from its own nativeLibraryDir (allowed for untrusted_app) on every boot, reading /data/local/tmp/mesh-node.conf (role/layers/peers/...) and the world-readable knot in place; holds an app PARTIAL_WAKE_LOCK (no root) and auto-detects the wlan0 IP (DHCP-safe). VERIFIED on R1 919109A4H16000691813: app reads the knot, binds the LAN port, serves correct inference (argmax 12095 @16.82); rebooted the entry node and it rejoined autonomously in ~12 s, /chain -> 12095, zero manual intervention. The mesh node now lives in the device's own OS surface, not as an external root-managed binary. REMAINING (polish, not blocking): reproducible APK packaging of the binary + extractNativeLibs, equip.sh writing mesh-node.conf, remove the dead Rust su hook.

• Research line: protocol69 / teleport for mesh (docs/R1_MESH_PROTOCOL69.md): FOIL-over-RF backing (a) RF discovery beacon as a WiFi-partition-tolerant liveness channel feeding stream-2010 gossip, and (b) cache-key teleport handoff — a pre-flight /.aeon/teleport-admit probe before each /chain forward that skips recompute on cache-warm hops.

  • (a) RF-gossip bridge — BUILT + VERIFIED (everything but the SDR front-end): src/rf_beacon.rs (Protocol69RfBeacon, fail-closed to_peer_info/to_gossip_frame)
    • bin/rf-gossip-bridge (selftest/file sources; rtlsdr a gated stub) + the mesh_gossip peer_heartbeat_frame/decode_heartbeat_frame injection API. Live test: a phantom beacon entered a running station's /mesh/peers; a forged envelope (expectedProjection:68) was dropped fail-closed. Standalone — no fat-station.rs change. REMAINING: the RTL-SDR demod front-end, gated on the R1 USB-OTG probe.
  • (b) cache-key teleport handoff — BUILT + VERIFIED (opt-in --teleport-cache): each /chain hop derives a prompt hash (fnv1a_64_tokens, propagated as X-Prefix-Hash), and on an identical PREFILL prefix replays its cached forward output (protocol69-gated, geodesicLength:0) instead of recomputing — a server-side guard in handle_chain (no residual round-trips back to the caller). Bit-exact (store/replay of the real residual; shape-checked), prefill-only (start_pos==0). Verified on a 2-node local chain: run1 543 ms argmax 12095 @16.82; run2 (both hops teleport) 17 ms argmax 12095 @16.82 BIT-IDENTICAL — 32.6x. Off by default = zero risk to the proven path. REMAINING: cache TTL/eviction (grows unbounded today).

• Fit-aware election -> data-parallel "both" lanes (2026-05-24): elect_stage_from_gossip now checks whether the model fits the node's RAM (model size <= 60% of MemAvailable). When it does — qwen-0.5b is 373 MB on ~2.7 GB-free nodes — the node elects role=both and serves the WHOLE model (layers 0-24) as one lane instead of taking a pipeline stage. This dissolves the entry-bound bottleneck: the old split made stage-0 the cap (one entry feeding many exits); now N nodes = **N independent full-model lanes**, the path to >=Nx throughput (cf. the cannon + MeshBatchSpeedup.lean). Only a model too big for one node falls back to the 0..12/12..24 split election.

• Gossip node health (jam awareness) — DONE (2026-05-24): every node advertises a compact NodeStatus (free/total RAM, 1-min load, in-flight request count, battery, temp, derived jammed) on the gossip heartbeat (PeerInfo.status, #[serde(default)] so it's wire-compatible with pre-status peers). A router drop-guard counts DATA-plane requests in flight (survives handler panics — the truest jam signal); a 1.5 s sampler refreshes the advertised health so each heartbeat is fresh. next_stage_pool routes AROUND jammed peers (falling back to them only if every replica of the next stage is jammed). GET /status exposes the same snapshot for a direct poll. So the whole mesh — routing, election, the 2002 brain — sees who is saturated without polling each node. (In-app nodes: battery/temp read "unknown" under SELinux; RAM/load/in-flight are live.)

• FLEET REFRESH — 5 nodes live, all both (2026-05-24): refreshed all live R1s to the gossip-NodeStatus build (USB, one at a time, equip.sh --tcpip --apk), each re-electing to role=both. Node ids are now hardware-hash (r1-<bitwise hash of ANDROID_ID>; see R1_FLEET.md). Power config flipped to screen-sleeps-but-node-stays-active (short screen timeout + WifiLock + no-doze; saves heat/throttle) and adb-over-wifi (:5555) is enabled so released nodes stay manageable. Verified: all 5 /status healthy, role=both, gossip-status propagating peer-to-peer (a refreshed node sees another's live free_mb).

• Next: the RTL-SDR front-end + on-R1 NESDR USB-OTG probe (RF path); flash the 6th R1; the live >=5x throughput benchmark (point aeon wall at the 5 both lanes); persistent adb-over-wifi (persist.adb.tcp.port, survives reboot — needs root); teleport cache TTL/eviction for production.

Benchmarks

Reproduce: scripts/r1/mesh-bench.py --entries <ip1>,<ip2> (latency cold/warm + sequential vs concurrent throughput). ALWAYS label cold vs warm AND power state (battery-released devices throttle vs plugged-in).

2026-05-24 — 4 nodes, fully redundant (entry x2, exit x2), all battery-released on WiFi (qwen-0.5b, stage plan 0..12/12..24, 5-token prefill -> logits, all argmax 12095):

• Battery-released latency is ~4-5x the earlier plugged ~766 ms 2-R1 figure — MediaTek throttles clocks off-charger (the wakelock keeps it awake, not fast). Keep the room's R1s charging if latency matters.
• The 3.23x exceeds the 2x you'd expect from two entry->exit pipelines alone, because each node also BATCHES concurrent requests (NATIVE_BATCH_CAPACITY=8), amortizing the per-request fixed cost. Formalized: gnosis-math/Gnosis/MeshBatchSpeedup.lean (pipeline parallelism P + per-node batching with positive fixed overhead -> speedup > P).

2026-05-24 — 5 nodes, all both (full-model data-parallel lanes), battery-released (the fit-aware-election topology; mesh-bench.py --entries <5 ips> --n 20, all argmax 12095):

• 5.66x clears the >=5x target — 5 independent full-model lanes plus per-node batching. Switching from the entry/exit split to fit-aware both removed the single-entry cap (the old 4-node split topped out at 3.23x); now the speedup scales with node count again.
• Warm latency roughly HALVED vs the split pipeline (~1.7 s vs ~3.8-4.5 s, both battery-released) because a both node serves the whole model locally — no WiFi relay hop in the data path. Off-charger MediaTek throttle still dominates absolute latency; keep the room charging for speed.