Sakana AI Introduces PC-ALM to Train 1000-Layer Networks

Researchers at Sakana AI have developed a novel artificial intelligence training algorithm named PC-ALM, functioning as a layer-local training method designed to replace traditional backpropagation. This architecture addresses the inherent limitations of standard Predictive Coding (PC) methods, which traditionally suffer from credit signal degradation before reaching input layers in deep and narrow networks.

The foundational mechanism of Predictive Coding treats every hidden activation as an optimization variable, penalizing the squared mismatch between each layer's activation and predictions arriving from below. Consequently, inference operates as gradient descent on that specific energy landscape, while learning is driven by a Hebbian-like weight step. The challenge lies in supervision entering at the output and diffusing through sequential compromises, with Innocenti et al. demonstrating that this PC-BP gap peaks when width is smaller than depth.

During inference, PC-ALM alternates between two local steps: a primal gradient step on the activations, and a dual step ${\lambda }_i\leftarrow {\lambda }_i+\alpha r_i$ that accumulates prediction errors for each respective layer. The research team interprets this composite structure as a proportional-integral (PI) controller per layer, where the prediction error serves as the proportional term and the multiplier acts as the integral term. Setting $\alpha =\rho$ yields the classical method of multipliers.

Stock photo for illustration only, not from the actual event

The research team swept residual MLPs with widths and depths ranging from 8 to 128 on Fashion-MNIST and MNIST datasets under a mean-field parameterization for 1 epoch. Utilizing an inference budget of $T=2L$, PC-ALM matched backpropagation performance across every width, depth, and activation function (Identity, tanh, ReLU), whereas standard PC exhibited sharp performance drops in deep and narrow cells.

"The repository's reference cell (width 32, depth 32, ReLU, Fashion-MNIST) reports 78.66% test accuracy for backpropagation, 68.13% for PC, and 77.75% for PC-ALM, with gradient cosine relative to backpropagation rising from 0.604 to 0.909."

Sakana AI Research Team

The study further expands this scope to evaluate 1000-layer residual MLPs on MNIST for 5 epochs, maintaining results within roughly two points of backpropagation. Furthermore, PC-ALM consistently outperformed standard PC across every evaluated benchmark, including ResNet-18 tested on CIFAR-10 and Tiny ImageNet.

Source: MarkTechPost