FFN Optimization Suite — Troubleshooting Guide

Status: Production support handbook
Date: May 18, 2026
Scope: Common issues, diagnosis, and resolution for deployment & operations

Quick Reference by Symptom

Symptom	Root Cause	Solution	Severity
Latency increased (not 1.8x)	Optimization disabled or misconfigured	Check `use_fused_gate_up` flag	🔴 Critical
NaN/Inf in outputs	Numerical instability in saturation	Disable saturation bitmasks	🔴 Critical
OOM (out of memory)	KV cache compression overhead	Reduce compression ratio	🔴 Critical
MMLU accuracy < 57.4%	Accuracy loss exceeds threshold	Disable LoRA++ or KV compression	🟡 High
Cache hit rate < 90%	Bitmask encoding mismatch	Regenerate saturation maps	🟡 High
Service won't start	.rknot metadata missing/corrupt	Re-encode .rknot files	🟡 High
Training slower than baseline	Spectral analysis overhead	Normal (one-time cost)	🟢 Low
Timeout on specific models	GPU memory insufficient	Scale up or reduce batch size	🟡 High

Category 1: Latency & Performance Issues

Problem 1.1: Latency NOT Improved (Stuck at Baseline)

Symptom:

Expected: 39.68 ms/token (1.89x)
Actual:   74.88 ms/token (1.00x, no improvement)

Root cause: Fusion or saturation kernel not active.

Diagnosis:

# Check config flags
grep -A 5 "use_fused_gate_up\|use_saturation_masks" /etc/inference/config.yaml
# Expected: true, true

# Check service logs
tail -100 /var/log/inference.log | grep -E "fusion|saturation"
# Expected: "fusion=ACTIVE" and "saturation=ACTIVE"

# Verify .rknot metadata loaded
strings /opt/models/phi3.rknot | grep "SaturationMaps"
# Expected: metadata signature found

Solutions (in order):

Re-enable flags:

kubectl set env deployment/inference-service \
  USE_FUSED_GATE_UP=true \
  USE_SATURATION_MASKS=true
kubectl rollout restart deployment/inference-service

Verify .rknot files:

# Check first 4 bytes (magic number)
xxd -l 4 /opt/models/phi3.rknot
# Expected: 4b 4e 4f 54 (KNOT)

# If corrupted, regenerate:
./target/release/encode-and-upload-rknots \
  --model phi3 \
  --saturation-maps /opt/cache/saturation-maps/phi3.rs \
  --knot-input /opt/models/phi3.rknot

Restart inference service:

systemctl restart inference-service
# Wait 30 seconds for warmup

Test fix:

python scripts/bench_latency.py --model phi3 --tokens 32 --check-speedup
# Expected: speedup >= 1.8x

Problem 1.2: Speedup Degradation Over Time

Symptom:

Hour 1: 1.89x speedup ✅
Hour 2: 1.75x speedup ⚠️
Hour 4: 1.42x speedup 🔴

Root cause: Cache memory fragmentation or thermal throttling.

Diagnosis:

# Check GPU thermal
nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader
# Expected: < 80°C, if > 85°C = thermal throttling

# Check memory fragmentation
nvidia-smi --query-gpu=memory.used,memory.free --format=csv,noheader
# Expected: free memory > 10% of total

# Check CPU thermal
watch -n 5 'sensors | grep Core'
# Expected: < 85°C

Solutions:

If thermal throttling:

# Scale down batch size
kubectl set env deployment/inference-service \
  MAX_BATCH_SIZE=8  # Reduce from 16

# Or reduce token sequence length
MAX_TOKENS=128  # Reduce from 256

If memory fragmentation:

# Restart service (clears GPU memory)
kubectl rollout restart deployment/inference-service

# Wait for convergence (5 min)
sleep 300

Check for memory leak:

# Monitor memory over 30 min
for i in {1..30}; do
  nvidia-smi --query-gpu=memory.used --format=csv,noheader
  sleep 60
done | tee memory_log.txt

# Plot to check for linear growth
# If memory grows > 1% per hour = memory leak

Test fix:

python scripts/bench_latency.py --model phi3 --duration 3600 --plot
# Expected: flat line at ~1.89x for full hour

Problem 1.3: Latency Spikes (p99 > 100ms)

Symptom:

p50: 39.68 ms ✅
p95: 42.1 ms ✅
p99: 156.3 ms 🔴 (expected < 50ms)

Root cause: Garbage collection, context switching, or cache contention.

Diagnosis:

# Check if coincides with GC
grep "gc\|garbage" /var/log/inference.log | grep -v "^#"
# Expected: minimal GC activity

# Check CPU utilization
top -b -n 1 | grep -E "CPU|inference" | head -5
# Expected: < 80% on single core (leave room for OS)

# Check context switches
pidstat -w -p $(pgrep -f inference-service) 1 5
# Expected: cswch/s < 100 (lower is better)

# Check I/O
iostat -x 1 5
# Expected: %await < 5% (no I/O bottleneck)

Solutions:

If GC spikes:

# Increase heap size (Rust doesn't have heap, but check allocator)
# For Python inference servers:
PYTHONUNBUFFERED=1 \
MALLOC_TRIM_THRESHOLD_=131072 \
./inference-service --bind 0.0.0.0:8080

If context switching:

# Reduce thread pool
NUM_WORKER_THREADS=4  # Reduce from 8

# Pin threads to cores
taskset -p -c 0-3 $(pgrep -f inference-service)

If cache contention:

# Reduce concurrent requests
MAX_CONCURRENT=4  # Reduce from 8

# Batch requests to warm cache
BATCH_SIZE=4  # Process multiple tokens together

Test fix:

python scripts/load_test.py \
  --model phi3 \
  --concurrent 10 \
  --duration 300 \
  --plot-latency
# Expected: p99 < 50ms sustained

Category 2: Numerical Stability & Correctness

Problem 2.1: NaN/Inf in Model Output

Symptom:

[inference] forward pass produced NaN at layer 13
[inference] NaN detected: downstream predictions invalid

Root cause: Saturation bitmask encoding error or weight matrix corruption.

Diagnosis:

# Inspect bitmask encoding
xxd /opt/cache/saturation-maps/phi3.rs | head -20
# Expected: valid Rust code starting with `pub const`

# Check for encoding errors
grep -n "INVALID\|ERROR\|0xFFFFFFFF" /opt/cache/saturation-maps/phi3.rs
# Expected: no matches

# Validate against source model
./target/release/verify-bitmask-encoding \
  --model phi3 \
  --bitmask-file /opt/cache/saturation-maps/phi3.rs
# Expected: all checksums match

Solutions:

Immediately disable saturation:

kubectl set env deployment/inference-service \
  USE_SATURATION_MASKS=false
kubectl rollout restart deployment/inference-service

Regenerate bitmasks:

# Delete old bitmask
rm /opt/cache/saturation-maps/phi3.rs

# Regenerate with more calibration tokens
./target/release/profile-saturation-bitmasks \
  --model phi3 \
  --calibration-tokens 2000  # Increase from 500
  --output /opt/cache/saturation-maps/phi3.rs

# Verify encoding
./target/release/verify-bitmask-encoding \
  --model phi3 \
  --bitmask-file /opt/cache/saturation-maps/phi3.rs

Re-encode .rknot:

./target/release/encode-and-upload-rknots \
  --model phi3 \
  --saturation-maps /opt/cache/saturation-maps/phi3.rs \
  --knot-input /opt/models/phi3.rknot

# Restart service
kubectl rollout restart deployment/inference-service

Test fix:

python scripts/check_numerical_stability.py \
  --model phi3 \
  --tokens 256 \
  --check-nan-inf
# Expected: no NaN/Inf over 256 tokens

Problem 2.2: Cosine Similarity Drop (< 0.99)

Symptom:

Cosine similarity: 0.95 (expected >= 0.99)
Meaning: optimized output differs from baseline by 5%

Root cause: Saturation threshold too aggressive or KV cache compression too extreme.

Diagnosis:

# Check saturation thresholds
grep "saturation_threshold\|frozen_neuron_ratio" /etc/inference/config.yaml
# Expected: saturation_threshold = 2 sigma (typical)

# Run per-layer cosine check
./target/release/bench-saturation-e2e \
  --model phi3 \
  --tokens 32 \
  --check-cosine
# Expected: per-layer cosine >= 0.98

# Check which layers have low cosine
tail -100 /var/log/inference.log | grep "cosine\|layer"

Solutions:

Reduce saturation aggression:

# Increase saturation threshold (freeze fewer neurons)
SATURATION_THRESHOLD=3.0  # Increase from 2.0
kubectl set env deployment/inference-service \
  SATURATION_THRESHOLD=3.0

Reduce KV cache compression:

# Lower compression ratio
kubectl set env deployment/inference-service \
  KV_COMPRESSION_RATIO=0.70  # Up from 0.50

Disable problematic optimizations:

# If Llama-70B: disable KV compression (known fragile)
kubectl set env deployment/inference-service \
  KV_COMPRESSION_ENABLED=false --selector model=llama

Test fix:

python scripts/verify_cosine_similarity.py \
  --model phi3 \
  --baseline baseline_outputs.pkl \
  --optimized optimized_outputs.pkl \
  --min-cosine 0.99
# Expected: all layers >= 0.99

Category 3: Memory & Resource Issues

Problem 3.1: Out of Memory (OOM)

Symptom:

CUDA out of memory: tried to allocate 24.6 MB on device 0
(GPU has 24 GB total, 0.2 GB free)

Root cause: KV cache compression overhead or batch size too large.

Diagnosis:

# Check GPU memory before crash
nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader
# Expected: memory.used / memory.total < 90%

# Monitor memory growth during inference
watch -n 1 'nvidia-smi | grep Processes -A 5'
# Expected: stable or slowly decreasing memory

# Check batch size
grep "batch_size\|max_batch" /etc/inference/config.yaml
# Expected: reasonable for model size (e.g., batch_size=4 for Llama-70B)

Solutions (in order):

Reduce batch size immediately:

kubectl set env deployment/inference-service \
  MAX_BATCH_SIZE=2  # Reduce significantly
kubectl rollout restart deployment/inference-service

Disable KV cache compression:

kubectl set env deployment/inference-service \
  KV_COMPRESSION_ENABLED=false

Scale down to smaller models:

# For Llama-70B: switch to Gemma-31B or Qwen-7B
kubectl patch deployment inference-service \
  -p '{"spec":{"template":{"spec":{"containers":[{"name":"inference","env":[{"name":"DEFAULT_MODEL","value":"gemma4-31b"}]}]}}}}'

Upgrade GPU memory:

# Last resort: move to machine with more VRAM
kubectl cordon node-with-24gb-gpu
kubectl uncordon node-with-40gb-gpu

Test fix:

python scripts/memory_stress_test.py \
  --model phi3 \
  --batch_size 4 \
  --max_tokens 2048
# Expected: no OOM, memory < 90% utilization

Problem 3.2: Saturation Bitmask Memory Overhead

Symptom:

Expected model size: 8.2 GB
Actual size with saturation: 8.3 GB
Overhead: 100 MB (unexpected, should be < 1 MB)

Root cause: Duplicate bitmask storage or inefficient encoding.

Diagnosis:

# Check .rknot file size
ls -lh /opt/models/phi3.rknot
# Expected: < 100 MB overhead for saturation metadata

# Inspect .rknot structure
./target/release/inspect-rknot \
  --file /opt/models/phi3.rknot \
  --show-sizes
# Expected: saturation section < 1 KB per layer

# Check for duplicate entries
cargo test --release test_rknot_no_duplicates -- --nocapture
# Expected: 0 duplicates found

Solutions:

Re-encode with compression:

./target/release/encode-and-upload-rknots \
  --model phi3 \
  --saturation-maps /opt/cache/saturation-maps/phi3.rs \
  --knot-input /opt/models/phi3.rknot \
  --compress-metadata

Validate .rknot integrity:

./target/release/validate-rknot \
  --file /opt/models/phi3.rknot
# Expected: "validation passed" message

Clear and regenerate:

# Backup original
cp /opt/models/phi3.rknot /opt/models/phi3.rknot.backup

# Regenerate from base model
./target/release/encode-and-upload-rknots \
  --model phi3 \
  --saturation-maps /opt/cache/saturation-maps/phi3.rs \
  --knot-input /opt/models/phi3-base.rknot

Test fix:

ls -lh /opt/models/*.rknot
# Expected: overhead < 1% of model size

Category 4: Cache & Coherence Issues

Problem 4.1: Cache Hit Rate Too Low (< 90%)

Symptom:

Saturation bitmask cache hit rate: 65.2% (expected > 95%)
Performance impact: 1.45x instead of 1.89x

Root cause: Bitmask encoding mismatch or incompatible layer indices.

Diagnosis:

# Check cache hit/miss metrics
prometheus_query 'rate(saturation_cache_misses_total[5m])'
# Expected: < 5% misses

# Analyze which layers are missing
tail -1000 /var/log/inference.log | \
  grep "cache_miss" | \
  awk '{print $NF}' | sort | uniq -c | sort -rn
# Expected: all layers present

# Verify bitmask layer coverage
./target/release/check-bitmask-coverage \
  --model phi3 \
  --bitmask-file /opt/cache/saturation-maps/phi3.rs
# Expected: coverage = 100% (32/32 layers)

Solutions:

Invalidate and rebuild cache:

# Clear in-memory cache
redis-cli -h $(hostname) -p 6379 FLUSHDB

# Restart service
kubectl rollout restart deployment/inference-service

# Warm cache with calibration data
python scripts/warm_saturation_cache.py \
  --model phi3 \
  --samples 100

Regenerate bitmasks with full coverage:

# Delete and regenerate
rm /opt/cache/saturation-maps/phi3.rs

# Use more calibration data
./target/release/profile-saturation-bitmasks \
  --model phi3 \
  --calibration-tokens 5000  # Increase from 500
  --output /opt/cache/saturation-maps/phi3.rs

Verify layer indices match:

# Check model layer count
grep -o "layers: [0-9]*" /etc/inference/config.yaml
# Expected: 32 for Phi-3-mini

# Check bitmask file header
head -5 /opt/cache/saturation-maps/phi3.rs | grep "layers"
# Expected: should match model config

Test fix:

python scripts/measure_cache_hit_rate.py \
  --model phi3 \
  --duration 300 \
  --target-hit-rate 0.95
# Expected: hit rate > 95% sustained

Problem 4.2: Bitmask Encoding Mismatch

Symptom:

[error] bitmask encoding mismatch: layer 13
[error] expected neuron_count=3072, got 3068
[error] cache incoherent: falling back to baseline

Root cause: .rknot metadata doesn't match actual model weights.

Diagnosis:

# Check model dimensions
./target/release/inspect-model \
  --file /opt/models/phi3.rknot \
  --show-layer 13
# Expected: hidden_dim=3072, intermediate=8192

# Check bitmask dimensions
./target/release/inspect-bitmask \
  --file /opt/cache/saturation-maps/phi3.rs \
  --show-layer 13
# Expected: neuron_count=3072

# Verify checksum match
md5sum /opt/models/phi3-base.rknot
./target/release/checksum-rknot \
  --file /opt/models/phi3.rknot \
  --compare-baseline
# Expected: checksums match

Solutions:

Force re-encode with strict validation:

./target/release/encode-and-upload-rknots \
  --model phi3 \
  --saturation-maps /opt/cache/saturation-maps/phi3.rs \
  --knot-input /opt/models/phi3.rknot \
  --strict-validation \
  --verbose

Validate .rknot against model weights:

# Extract .rknot metadata and compare to weights
cargo test --release test_rknot_layer_dimension_match -- --nocapture
# Expected: 0 mismatches

Fallback to unoptimized .rknot:

# Restore backup
cp /opt/models/phi3.rknot.backup /opt/models/phi3.rknot

# Disable saturation until fixed
kubectl set env deployment/inference-service \
  USE_SATURATION_MASKS=false

Test fix:

./target/release/smoke-test-saturation --model phi3 --tokens 32
# Expected: "bitmask validation passed"

Category 5: Model Loading & Initialization

Problem 5.1: .rknot File Won't Load

Symptom:

[error] knot magic mismatch: expected 'KNOT', got 'HTML'
[error] failed to load model: corrupted .rknot file

Root cause: File download corruption or wrong file format.

Diagnosis:

# Check file magic number
xxd -l 16 /opt/models/phi3.rknot
# Expected: first 4 bytes = 4b 4e 4f 54 (KNOT)

# Check file size (sanity check)
ls -lh /opt/models/phi3.rknot
# Expected: ~8 GB (reasonable for 3.8B model)

# Check if HTML error page
head -c 100 /opt/models/phi3.rknot | strings
# Expected: no HTML tags, should see binary data

# Verify download checksum
sha256sum /opt/models/phi3.rknot
# Compare to known good checksum from inventory

Solutions:

Re-download .rknot file:

# Remove corrupted file
rm /opt/models/phi3.rknot

# Re-download from repository
gsutil cp gs://forkjoin-models/phi3-mini.rknot /opt/models/phi3.rknot

# Verify checksum
sha256sum /opt/models/phi3.rknot | grep -f /opt/models/phi3.rknot.sha256
# Expected: match found

If download network issues:

# Try with retry logic
wget --tries=5 --retry-connrefused \
  https://models.forkjoin.ai/phi3-mini.rknot \
  -O /opt/models/phi3.rknot

Rebuild .rknot from base weights:

# If original weights available
./target/release/encode-and-upload-rknots \
  --model phi3 \
  --weights-file /opt/weights/phi3-base.safetensors \
  --output /opt/models/phi3.rknot

Test fix:

./target/release/smoke-test-saturation --model phi3 --tokens 1
# Expected: "model loaded successfully"

Problem 5.2: Metadata Fields Missing

Symptom:

[warn] .rknot metadata incomplete: saturation_maps missing
[warn] falling back to standard FFN (no speedup)

Root cause: .rknot encoded without saturation metadata.

Diagnosis:

# Check if metadata field present
strings /opt/models/phi3.rknot | grep -i "saturation\|metadata"
# Expected: "SaturationMaps" or similar found

# Use binary inspection tool
./target/release/inspect-rknot \
  --file /opt/models/phi3.rknot \
  --show-metadata
# Expected: metadata section with saturation_maps

Solutions:

Re-encode with metadata:

./target/release/encode-and-upload-rknots \
  --model phi3 \
  --saturation-maps /opt/cache/saturation-maps/phi3.rs \
  --knot-input /opt/models/phi3.rknot \
  --include-metadata

Add metadata post-hoc:

./target/release/add-rknot-metadata \
  --file /opt/models/phi3.rknot \
  --saturation-maps /opt/cache/saturation-maps/phi3.rs

Validate metadata persisted:

./target/release/inspect-rknot \
  --file /opt/models/phi3.rknot \
  --verify-metadata
# Expected: "metadata validation passed"

Test fix:

./target/release/smoke-test-saturation --model phi3 --tokens 32 --check-metadata
# Expected: "saturation metadata loaded"

Category 6: LoRA++ Fine-Tuning Issues

Problem 6.1: LoRA++ Training Slower Than Baseline

Symptom:

Expected: 7.4 hours (40.6% memory reduction)
Actual: 8.9 hours (slower than baseline 8.2 hours)

Root cause: Spectral analysis overhead (should be amortized).

Diagnosis:

# Check if spectral analysis per-training
grep "spectral_analysis\|cliff_detection" training.log
# Expected: "spectral analysis" appears once at startup, not per epoch

# Check LoRA training loop
python -c "
import time
model = load_model('phi3')
start = time.time()
optimizer.step()
end = time.time()
print(f'per-step overhead: {(end-start)*1000:.2f}ms')
"
# Expected: < 1ms per step overhead

Solutions:

Verify spectral analysis is cached:

# Check cache directory
ls -lh /opt/cache/spectral-atlas/phi3.atlas
# Expected: file exists and is recent (< 1 day old)

# If missing, regenerate once
./target/release/load-spectral-atlas \
  --model phi3 \
  --output /opt/cache/spectral-atlas/phi3.atlas

Use precomputed cliff scores:

# In training script
from gnosis.lora_plus_plus import load_cliff_scores

cliff_scores = load_cliff_scores("phi3")  # Load from cache
lora_config = allocate_cliff_guided_ranks(cliff_scores)
# This should be ~0 overhead (O(1) lookup)

Profile training loop:

python -m cProfile -s cumtime training_script.py > profile.txt
head -30 profile.txt
# Check if spectral analysis shows high cumtime

Explanation: Spectral analysis is a one-time cost (< 5 min). Training time might appear slower due to measurement variability. Run 3 trials and take median.

Problem 6.2: LoRA++ Accuracy Loss > 0.7%

Symptom:

Baseline MMLU: 58.2%
With LoRA++: 56.8% (1.4% loss, exceeds 0.7% threshold)

Root cause: Rank allocation too aggressive (allocated ranks too low).

Diagnosis:

# Check allocated ranks
./target/release/inspect-lora-config \
  --model phi3 \
  --config config.yaml
# Expected: fragile=32, valley=8, refinement=16

# Verify cliff scores loaded correctly
python -c "
from gnosis.lora_plus_plus import load_cliff_scores
scores = load_cliff_scores('phi3')
for layer, score in enumerate(scores[:5]):
    print(f'layer {layer}: sigma1/sigma2 = {score:.2f}')
"
# Expected: early layers < 8 (fragile), middle > 8 (compressible)

# Compare rank allocation to baseline
grep -A 20 "rank_allocation" training.log | head -15
# Expected: no layers allocated rank=1

Solutions:

Increase fragile layer ranks:

# In lora_plus_plus config
cliff_guided_ranks = {
    'fragile_rank': 48,   # Increase from 32
    'valley_rank': 8,     # Keep low
    'refinement_rank': 20  # Increase from 16
}

Reduce compression ratio (fallback):

# If accuracy critical
cliff_guided_ranks = {
    'fragile_rank': 32,
    'valley_rank': 16,    # Increase from 8
    'refinement_rank': 24  # Increase from 16
}
# Trade: 40.6% memory savings → 30% savings, but better accuracy

Validate on MMLU before full training:

python scripts/validate_lora_config.py \
  --model phi3 \
  --config lora_config.yaml \
  --mmlu_samples 50 \
  --acceptable_loss 0.7
# Expected: "config acceptable" message

Test fix:

python scripts/lora_plus_plus_validation.py \
  --model phi3 \
  --config lora_config.yaml \
  --check-mmlu
# Expected: loss < 0.7%

Category 7: Deployment & Rollout Issues

Problem 7.1: Canary Rollout Stuck

Symptom:

Canary pool (20%): latency 1.89x ✅
Full pool (100%): latency 1.42x 🔴
Traffic won't shift beyond canary

Root cause: Load balancer misconfiguration or pool health check failing.

Diagnosis:

# Check load balancer status
kubectl get svc inference-service -o yaml | grep -A 10 "endpoints:"
# Expected: 3 endpoints for full pool

# Check pod health
kubectl get pods -l app=inference-service -o wide
# Expected: all pods "Running" and "Ready"

# Check health check endpoint
curl -s http://[pod-ip]:8080/health
# Expected: HTTP 200, {"status": "ok"}

Solutions:

Fix health check endpoint (if failing):

# Check service logs
kubectl logs -f deployment/inference-service | grep -i health

# Restart pods
kubectl rollout restart deployment/inference-service

Manually shift traffic:

# If health checks stuck, manually patch service
kubectl patch service inference-service \
  -p '{"spec":{"selector":{"app":"inference-service"}}}'

# Verify traffic distribution
kubectl describe svc inference-service

Increase health check timeout:

# Health check too strict
kubectl patch deployment inference-service \
  --type='json' \
  -p='[
    {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds", "value": 60},
    {"op": "replace", "path": "/spec/template/spec/containers/0/livenessProbe/timeoutSeconds", "value": 10}
  ]'

Test fix:

# Verify all backends responsive
for pod in $(kubectl get pods -o name); do
  kubectl exec -it $pod -- curl -s localhost:8080/health
done
# Expected: all return 200

Problem 7.2: Rollback Incomplete (Mixed Versions)

Symptom:

Some pods running optimized version (1.89x)
Some pods running baseline (1.00x)
Inconsistent latency metrics

Root cause: Rollback command didn't wait for convergence.

Diagnosis:

# Check pod versions
kubectl get pods -l app=inference-service -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
# Expected: all same image hash

# Check deployment status
kubectl rollout status deployment/inference-service
# Expected: "rollout successful"

Solutions:

Force restart all pods:

kubectl rollout restart deployment/inference-service
kubectl rollout status deployment/inference-service --timeout=5m

Verify convergence:

# Wait for all pods Ready
kubectl wait --for=condition=Ready pod \
  -l app=inference-service \
  --timeout=300s

Check metrics consistency:

# All pods should report same speedup
promtool query instant 'inference:speedup_ratio'
# Expected: all values >= 1.8 or all values == 1.0, no mix

Test fix:

python scripts/validate_rollout_consistency.py \
  --check-latency \
  --check-version
# Expected: "all pods consistent"

Emergency Procedures

Immediate Disable All Optimizations

Use in case of unexpected issues:

# Single command to disable everything
kubectl set env deployment/inference-service \
  USE_FUSED_GATE_UP=false \
  USE_SATURATION_MASKS=false \
  USE_KV_COMPRESSION=false

# Restart immediately
kubectl rollout restart deployment/inference-service

# Verify reverted to baseline
sleep 30
curl -s http://inference:9090/metrics | grep speedup
# Expected: speedup ~ 1.0x

Timeline: < 2 minutes to complete revert

Full Deployment Rollback

If critical issue detected:

# Step 1: Immediately halt traffic
kubectl patch service inference-service -p '{"spec":{"selector":{"app":"inference-service-old"}}}'

# Step 2: Rollback deployment
kubectl rollout undo deployment/inference-service
kubectl rollout status deployment/inference-service

# Step 3: Verify old version running
kubectl get deployment inference-service -o jsonpath='{.spec.template.spec.containers[0].image}'
# Expected: old image hash

# Step 4: Run smoke test
./target/release/smoke-test-saturation --model phi3 --tokens 32
# Should run on baseline (no speedup expected)

Timeline: < 5 minutes

Support & Escalation

On-Call Contact

Component	Owner	Slack	PagerDuty
FFN Optimization	Taylor	#inference-ops	@inference-oncall
Infrastructure	DevOps team	#infrastructure	@infra-oncall
ML/Models	ML team	#ml-infra	@ml-oncall

Escalation Path

Severity 🔴 (Critical): Any production incident
- Immediate page on-call
- Message #inference-ops
- Escalate to VP Engineering if no resolution in 15 min
Severity 🟡 (High): Degraded performance
- Message on-call
- Create P1 ticket
- Escalate within 30 min if no progress
Severity 🟢 (Low): Degradation or warnings
- Log ticket
- Investigate in next stand-up

Appendix: Quick Commands Reference

# Check status
kubectl get pods -l app=inference-service
kubectl logs -f deployment/inference-service | grep -i error

# Monitor metrics
watch -n 5 'curl -s http://inference:9090/metrics | grep -E "latency|speedup|error"'

# Test model loading
./target/release/smoke-test-saturation --model phi3 --tokens 32

# Benchmark latency
./target/release/bench-saturation-e2e --model phi3 --tokens 128

# Restart service
kubectl rollout restart deployment/inference-service

# Disable all optimizations
kubectl set env deployment/inference-service \
  USE_FUSED_GATE_UP=false \
  USE_SATURATION_MASKS=false \
  USE_KV_COMPRESSION=false

# Full rollback
kubectl rollout undo deployment/inference-service

Last Updated: May 18, 2026

Questions? Contact #inference-ops on Slack or email ops@forkjoin.ai

forkjoin-ai/gnosis