Neuromimetic Activations

1. Resonant Spiking Dynamics

Grounding in the Leaky Integrate-and-Fire (LIF) model with hierarchical resonance.

The internal potential $V$ at time $t$ integrates incoming stimulus $I(t)$ and decayed previous state: $V_{integrated}(t) = I(t) + V(t-1) \cdot (1 - \lambda)$

Variables:

• $I(t)$: Incoming stimulus (bottom-up input).
• $\lambda$: leak_rate (passive decay of membrane potential).
• $\rho$: Hierarchical resonance factor (top-down modulation).
• $\theta$: Firing threshold.

State Evolution: $V_{res}(t) = V_{integrated}(t) \cdot (1 + \rho)$ $S(t) = \mathrm{STE}\bigl(\mathbb{I}(V_{res}(t) > \theta)\bigr)$ $V(t) = V_{res}(t) - S(t) \cdot \theta$

The forward pass is the hard indicator. The backward pass uses a sigmoid with slope sharpness (straight-through estimator).

2. Homeostatic Activity Scaling

Synaptic scaling mechanism to maintain firing rates in optimal ranges.

The gain $G(t)$ is regulated by the deviation from the target activity $A_{target}$: $G(t) = \mathrm{clip}\bigl(1 + \eta \cdot (A_{target} - \bar{A}),\; 0.1,\; 10\bigr)$

Parameters:

• $\eta$: adaptation_rate (speed of homeostatic correction).
• $A_{target}$: Target firing rate (e.g., 0.12).
• $\bar{A}$: Moving average of historical activity.
• $\alpha$: Smoothing factor for activity tracking.

Activity Update: $\bar{A}(t) = (1 - \alpha) \cdot \bar{A}(t-1) + \alpha \cdot A(t)$ $y = \text{GELU}(x \cdot G(t))$

3. Adaptive Softplus (Biological Approximation)

A smooth, differentiable activation for biological firing rate mimicry.

Core Formula: $f(x, \theta, s) = \frac{\ln(1 + \exp(s \cdot (x - \theta)))}{s}$

Benefits:

• Sparsity: Naturally suppresses low-intensity noise.
• Optimization: Provides non-zero gradients even in the 'off' state.
• Flexibility: Sharpness $s$ can be tuned for different semantic granularities.

4. Inference-Time State Evolution

Neuromimetic activations shift from static mappings to dynamic state transitions.

ARCANE activations are stateful and contextual: $(y, V_t) = f(x, V_{t-1}, \text{context})$

Key Differences:

• Memory: Standard activations are stateless ($y = f(x)$).
• Adaptation: ARCANE activations can update their feature-wise state across calls.
• Resonance: Responses are modulated by global semantic alignment.