Resonance-Aware Learning Rate Tuning — Implementation Summary

Completed Components

1. ResonanceFrequencyCalculator (resonance_scheduler.py)

Purpose: Computes natural resonance frequency (ω₀) of a transformer model from its architecture.

Key Methods:

• compute_resonance_frequency(): Returns ω₀ in rad/step

  • Computes spectral condition number κ from architecture
  • Estimates effective mass m from hidden dimension
  • Applies formula: ω₀ = √(κ/m)
  • Clamped to [0.001, 10] rad/step

• compute_optimal_learning_rate(base_scale=1e-3, num_training_steps=100000): Returns recommended LR

  • Uses inverse scaling: lr_opt = base_scale / ω₀
  • Applies damping factor based on training length
  • Clamped to [1e-6, 1e-2]

• get_diagnostics(): Returns dict of computed metrics

  • Architecture parameters
  • Resonance frequency
  • Spectral condition number
  • Estimated L2 norm

Type Safety: Full type hints on all methods and parameters.

Error Handling:

• Validates hidden_dim, num_layers, attention_heads > 0
• Handles spectral ratio edge cases (prevents div by zero)
• Logs warnings for numerical instabilities

2. AdaptiveResonanceLRScheduler (resonance_scheduler.py)

Purpose: Dynamically adjusts learning rate based on spectral metrics (σ₁/σ₂ ratio) during training.

Key Methods:

• step(loss, spectral_ratios, spectral_diagnostics): Execute one scheduler step

  • Returns ResonanceMetrics with full diagnostics
  • Classifies training phase
  • Computes LR adjustment factor
  • Applies learning rate if optimizer is set

• set_optimizer(optimizer): Set or replace PyTorch optimizer

• export_metrics(filepath): Write JSON file with metrics history

• get_diagnostics(): Return summary statistics

Training Phases:

Cliff Detection:

• Monitors relative loss increase: (loss - last_loss) / abs(last_loss)
• Triggers on > cliff_threshold (default 20%)
• Cuts LR by 50%, enters recovery for 10-50 steps

Spectral Ratio Monitoring:

• Per-layer σ₁/σ₂ computed via SVD
• Tracks: mean, min, max, drift
• Default target: 2.0 (configurable)
• Tolerance: ±15% of target before entering DRIFT phase

Type Safety: Full type hints, @dataclass ResonanceMetrics.

Error Handling:

• Validates min_lr ≤ base_lr ≤ max_lr
• Validates target_sigma_ratio > 1.0
• Graceful handling when optimizer not set (logged warning)
• Bounds all LR adjustments to [min_lr, max_lr]

3. ResonanceMetrics (resonance_scheduler.py)

Data Structure: @dataclass containing per-step diagnostics

Fields:

timestamp: int              # Unix timestamp
step: int                   # Global step count
loss: float                 # Current loss value
current_lr: float           # Applied learning rate
mean_sigma_ratio: float     # Mean σ₁/σ₂ across layers
min_sigma_ratio: float      # Minimum σ₁/σ₂
max_sigma_ratio: float      # Maximum σ₁/σ₂
sigma_ratio_drift: float    # |σ₁/σ₂ - target|
phase: ResonancePhase       # Enum: INITIALIZATION|STABLE|DRIFT|CLIFF|RECOVERY
cliff_triggered: bool       # Whether cliff was detected
lr_adjustment_factor: float # Multiplier applied to LR
spectral_layer_diagnostics: Dict  # Per-layer SVD metrics

Methods:

• to_dict(): Serialize to JSON-compatible dict

4. compute_spectral_ratio() Utility

Purpose: Compute σ₁/σ₂ ratio for a PyTorch model.

Signature:

def compute_spectral_ratio(
    model: Optional[nn.Module] = None,
    layer_index: Optional[int] = None,
) -> Tuple[float, Dict[int, Dict[str, float]]]

Returns:

• mean_ratio: Overall mean σ₁/σ₂
• per_layer_diagnostics: Dict mapping layer_idx to:
  • sigma_1, sigma_2: Top 2 singular values
  • ratio: σ₁/σ₂
  • layer_name: Parameter name
  • weight_shape: Tensor dimensions

Implementation:

• Uses SVD via torch.linalg.svd or torch.svd_lowrank
• Handles 2D+ tensors (reshapes if needed)
• Graceful fallback on SVD errors
• Returns (2.0, {}) if model is None

Integration Points

Training Loop Integration

# 1. Compute optimal LR from architecture
calc = ResonanceFrequencyCalculator(3072, 32, 32, 8192)
optimal_lr = calc.compute_optimal_learning_rate(num_training_steps=100000)

# 2. Initialize optimizer with predicted LR
optimizer = optim.AdamW(model.parameters(), lr=optimal_lr)

# 3. Create scheduler
scheduler = AdaptiveResonanceLRScheduler(
    optimizer=optimizer,
    base_lr=optimal_lr,
    target_sigma_ratio=2.0,
    enable_resonance_tuning=True,
)

# 4. Training loop
for epoch in range(num_epochs):
    for batch in dataloader:
        loss = forward_pass(batch)
        optimizer.step()

        # Compute spectral ratio
        sigma_ratio, diag = compute_spectral_ratio(model)

        # Update scheduler
        metrics = scheduler.step(loss, [sigma_ratio] * num_layers, diag)

        # Optional: Log to W&B/TensorBoard
        # wandb.log(metrics.to_dict())

# 5. Export metrics
scheduler.export_metrics("metrics.json")

CLI Interfaces

1. Resonance Calculator Demo

File: resonance_scheduler.py

python optimizers/resonance_scheduler.py \
  --hidden-dim 3072 \
  --num-layers 32 \
  --attention-heads 32 \
  --target-sigma-ratio 2.0 \
  --enable-resonance-tuning \
  --output-json diag.json

Output: JSON with architecture diagnostics + simulated 1000-step training run

2. Full Training Example

File: training_integration_example.py

python optimizers/training_integration_example.py \
  --model phi3 \
  --batch-size 32 \
  --num-steps 1000 \
  --enable-resonance-tuning \
  --compute-spectral \
  --output-metrics metrics.json \
  --log-level INFO

Features:

• Synthetic transformer model (can replace with real model)
• Full training loop with resonance scheduling
• Metrics export to JSON
• Logging with multiple levels

File Structure

open-source/gnosis/distributed-inference/optimizers/
├── __init__.py                          # Package exports
├── resonance_scheduler.py               # Core implementation (900+ lines)
├── training_integration_example.py      # Full training loop example
├── test_resonance_scheduler.py          # Unit tests (10 passed)
├── README.md                            # Full user documentation
└── IMPLEMENTATION_SUMMARY.md            # This file

Code Quality

Type Hints

• ✓ All function signatures have type hints
• ✓ Return types specified
• ✓ Optional parameters clearly marked
• ✓ Dict/List generic types parameterized

Docstrings

• ✓ Comprehensive module docstring with examples
• ✓ Class docstrings with attributes and purpose
• ✓ Method docstrings with Args, Returns, Raises
• ✓ Inline comments for complex logic

Error Handling

• ✓ Input validation with descriptive error messages
• ✓ Graceful degradation (e.g., when torch unavailable)
• ✓ Warning logs for recoverable issues
• ✓ Exception handling in compute_spectral_ratio

Logging

• ✓ Module-level logger initialized
• ✓ INFO level for major events (phase transitions, LR changes)
• ✓ DEBUG level for detailed diagnostics
• ✓ WARNING level for instabilities
• ✓ ERROR level for failures

Testing

• ✓ 10 unit tests covering:
  • Architecture parameter validation
  • Resonance frequency computation
  • Learning rate prediction
  • Scheduler initialization
  • Phase transitions
  • Cliff detection
  • Metrics export
  • End-to-end workflow
• ✓ All tests pass
• ✓ Works with and without pytest

Performance Characteristics

Computational Complexity

• ResonanceFrequencyCalculator: O(1) — only arithmetic on scalars
• AdaptiveResonanceLRScheduler.step(): O(1) — simple numeric comparisons
• compute_spectral_ratio(): O(m² × min(m,n)) for m×n matrices via SVD
  • With torch.svd_lowrank: O(m×n) with low rank approximation

Memory Usage

• Scheduler history: O(num_steps) for metrics list
  • ~500 bytes per ResonanceMetrics object
  • 1000 steps ≈ 500 KB
• Per-layer diagnostics optional, only if passed to step()

Recommended Usage

• Compute spectral ratio every 10-100 steps (not every step)
• Export metrics every epoch
• Warmup_steps ≈ 10% of training steps

Validation

Tested Architectures

• Phi-3: hidden_dim=3072, num_layers=32, attention_heads=32
• Qwen-2.5-7B: hidden_dim=4096, num_layers=28, attention_heads=32
• Llama-70B: hidden_dim=4096, num_layers=80, attention_heads=32
• BERT-small: hidden_dim=768, num_layers=12, attention_heads=12

Example Output

Optimal learning rate: 1.36e-04 (ω₀=7.357077, damping=1.49)

For Phi-3 with 100K training steps:

• Resonance frequency: ~7.36 rad/step
• Recommended learning rate: ~1.36e-04 (vs. typical 1e-3 for baseline AdamW)

Dependencies

Required

• Python 3.8+
• numpy
• torch (for model training; optional for ResonanceFrequencyCalculator)

Optional

• pytest (for test suite)
• wandb (for logging, can be integrated)

Known Issues

• OpenMP duplicate initialization on macOS (workaround: export KMP_DUPLICATE_LIB_OK=TRUE)

Next Steps / Future Enhancements

1. Distributed Training: Extend to handle multi-GPU σ₁/σ₂ computation
2. Mixed Precision: Adapt spectral monitoring for fp16/bfloat16 training
3. Integration: Add hooks to Hugging Face Transformers, Pytorch-Lightning
4. Visualization: Web dashboard for real-time metrics monitoring
5. Automatic Tuning: Optimize target_sigma_ratio per-model via meta-learning
6. Torch Export: C++ implementation for production inference

References

• Spectral Analysis: Golub & Van Loan (1996) "Matrix Computations"
• LR Scheduling: Smith & Topin (2019) "Super-Convergence"
• Resonant Systems: Resonance frequency modeling from harmonic oscillator theory
• Implementation: Buley et al. (2026) "Resonance-Aware Learning Rate Tuning"

License

MPL-2.0 (Multi-Platform License)

Author

Taylor Buley (taylorbuley@gmail.com)