Hierarchical Neural Resonance

Overview

Hierarchical Neural Resonance aligns recurrent hidden states with per-example top-down projections before a System 1 decision is committed. The explicit predict_with_resonance path carries one target per example with shape (batch, units), so an example's target remains independent of the other examples in its batch. Learned weights stay fixed unless the separate test-time adapter or BCM fast weights are enabled.

ARCANE also provides objective-driven test-time adaptation. That path can update selected variables using entropy, consistency, reconstruction, masked-token, or custom objectives, with snapshot/reset semantics, bounded update norms, and rollback. State refinement and weight adaptation are distinct mechanisms and should be evaluated separately.

Why state alignment matters

Many models produce an output after one pass through a fixed computation graph. Recurrent refinement is useful to study because it permits a model to spend additional computation on the current input and expose an explicit disagreement signal between adjacent levels. This does not imply that ordinary feed-forward or Transformer models cannot reason, nor that recurrence automatically improves reasoning.

How hierarchical resonance works

ARCANE introduces bidirectional information flow inspired by predictive coding. Higher layers project targets downward (Feedback Projection), lower layers move toward those targets (Harmonization), and the same current input is evaluated again. The implementation retains the prediction from the cycle with the lowest measured divergence, giving the application a concrete selection rule for the refinement trace.

Feedback maps can be trained end to end with an explicit lower-state reconstruction/alignment loss. Identity initialization is a safe baseline when adjacent levels have equal width, but learned feedback should be used only after it receives a real gradient path.

To understand Hierarchical Neural Resonance, it is essential to define the key terminologies that govern its behavior:

1. Prospective Configuration

Prospective Configuration is the principle where neural activities (activations) are optimized to align with hierarchical expectations before any synaptic weight updates (learning) occur.

2. Neural Resonance

Neural Resonance is the iterative exchange of projected targets and recurrent states between layers. It allocates bounded computation to a direct System 1 prediction and exposes the resulting alignment trace.

3. Harmonization

Harmonization is the specific mechanism by which a layer adjusts its internal semantic representation to minimize divergence from top-down feedback signals.

4. Feedback Projection

Feedback Projection is the top-down signal sent from a higher-level layer to a lower-level layer representing higher context expectations.

5. Prediction Divergence

Prediction Divergence is the measured difference (error) between a layer's current internal semantic representation and the Feedback Projection it receives.

6. Adaptive Alignment Phase

Resonance phase is an explicit current-input loop run by predict_with_resonance. The legacy NeuralResonanceCallback only stages a previous-batch mean prototype before a training batch and is not equivalent to this loop.

7. Self-Modeling

Self-Modeling is the capability of a neural hierarchy to maintain and refine a persistent internal semantic representation of its own state.

The architecture of a Resonant model is hierarchical and bi-directional. While information flows upward during the forward pass, expectations flow downward during the resonance phase.

Implemented properties

• Per-example targets: explicit inference does not collapse the batch into one prototype.
• Measurable alignment: adjacent-level reconstruction error and state divergence are exposed directly.
• Controlled compute: cycle count is configurable and can be compared from the same checkpoint.
• Reversible adaptation: optional fast-weight updates can be bounded, rolled back, and reset after each request.

Disadvantages and Limitations

Despite its theoretical advantages, the resonant approach introduces specific trade-offs:

• Compute and latency: every resonance cycle adds another forward evaluation of the current input.
• Objective mismatch: lower alignment divergence need not improve accuracy, calibration, or generation quality.
• Adaptation risk: entropy minimization and BCM can reinforce confident errors or introduce order sensitivity.
• Evidence limits: mechanism tests establish implementation behavior, not task superiority or biological equivalence.

The ARCANE framework implements this theory through two primary components:

1. The ResonantGSER Layer

The fundamental unit of the resonance hierarchy:

• Closed-form harmonization: $N$ EMA steps toward resonance_alignment, skipped while that vector is unset.
• Feedback Mechanism: project_feedback() maps hidden→hidden by default; to_input_space=True uses feedback_weights.
• Per-example inference state: predict_with_resonance obtains (batch, units) sequence summaries and carries one target per example. last_h is retained only as a batch-mean compatibility summary for legacy APIs.
• Spikes: graded spikes, signed counts against a self-scaling threshold (graded_spike), with a learned per-unit threshold and leak.

2. The NeuralResonanceCallback

Runs on on_train_batch_begin (before the current batch forward pass):

1. Feedback Phase: Higher layers project from last_h (previous batch prototype).
2. Harmonization Phase: Lower layers write that projection into resonance_alignment.
3. Forward pass: The upcoming batch is attracted toward that prototype.

This callback is not an inner-loop re-forward of the current batch. Use predict_with_resonance for per-example hierarchical refinement, or PredictiveResonantLayer for local recurrent alignment.

ARCANE includes exploratory comparisons and a controlled same-checkpoint ablation harness. The published example numbers below are historical smoke tests, not comprehensive validation.

Model Architectures Compared

Tiny Shakespeare Benchmark (15,000 chars, 10 epochs)

These numbers are from a small exploratory run with unequal parameter counts. They are not a Transformer comparison and should only be treated as historical System 1 smoke-test results.

Training Dynamics Analysis

MNIST (Spatio-Temporal Digit Classification)

Key Findings (exploratory, small data)

• Adding resonance levels changed Tiny Shakespeare accuracy by about one point at roughly 2× parameters.
• MNIST 98.89% vs 98.76% is within noise for that setup.
• Unit tests verify closed-form harmonization, STE spikes, and Keras 3 layer linking, not task superiority.

The mechanism is inspired by predictive-coding and recurrent-processing ideas. ARCANE implements engineering analogies such as top-down projections, local divergence, spiking transforms, and homeostatic traces, but does not establish neuroscientific fidelity.

Hierarchical Neural Resonance combines per-example state, differentiable feedback training, bounded test-time objectives, and a same-checkpoint ablation harness. Together, these features make recurrent refinement measurable and let each application select the cycle depth that best serves its accuracy and latency requirements.

Empirical Validation

The existing Tiny Shakespeare and MNIST numbers are exploratory, use limited seeds or unequal parameter counts, and do not isolate inference-time learning. A proper evaluation compares 0/1/2/4/8 cycles on the same checkpoint over multiple seeds and reports accuracy, negative log-likelihood, calibration, latency, forgetting, order sensitivity, batch-size sensitivity, and distribution shift.

Future Directions

• Evaluate whether alignment objectives correlate with downstream quality.
• Compare learned feedback with tied and identity feedback.
• Test persistent adaptation under ordering and distribution shifts.
• Profile cycle latency and memory at realistic sequence lengths.