The Cascaded Forward Algorithm for Neural Network Training

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   Training convolutional neural networks with the Forward–Forward Algorithm

   Riccardo Scodellaro, A. Kulkarni, Frauke Alves et al. · 2023 · Scientific Reports · cited 27 times · Not summarized yet

   Abstract

   Recent successes in image analysis with deep neural networks are achieved almost exclusively with Convolutional Neural Networks (CNNs), typically trained using the backpropagation (BP) algorithm. In a 2022 preprint, Geoffrey Hinton proposed the Forward–Forward (FF) algorithm as a biologically inspired alternative, where positive and negative examples are jointly presented to the network and training is guided by a locally defined goodness function. Here, we extend the FF paradigm to CNNs. We introduce two spatially extended labeling strategies, based on Fourier patterns and morphological transformations, that enable convolutional layers to access label information across all spatial position…

2. Very close Not relevant

   Scalable Forward-Forward Algorithm

   Andrii Krutsylo · 2025 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Abstract

   We propose a scalable Forward-Forward (FF) algorithm that eliminates the need for backpropagation by training each layer separately. Unlike backpropagation, FF avoids backward gradients and can be more modular and memory efficient, making it appealing for large networks. We extend FF to modern convolutional architectures, such as MobileNetV3 and ResNet18, by introducing a new way to compute losses for convolutional layers. Experiments show that our method achieves performance comparable to standard backpropagation. Furthermore, when we divide the network into blocks, such as the residual blocks in ResNet, and apply backpropagation only within each block, but not across blocks, our hybrid des…

3. Very close Cites this paper Not relevant

   Exploring the Feasibility of Forward Forward Algorithm in Neural Networks

   Suraj R Gautham, S. Nair, Suresh Jamadagni et al. · 2024 · 2024 International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE) · cited 1 time · Not summarized yet

   Abstract

   This research explores the Forward-Forward Algorithm, proposed as an alternative to Backpropagation, in the context of Convolutional and Recurrent Neural Networks for multiclass classification. The authors' investigation aims to evaluate the algorithm's efficiency, advantages, and disadvantages for real-world applications. Results offer insights into the untapped potential and nuances of this approach, guiding researchers in optimizing neural network performance and potentially advancing a more human-like approach in Deep Learning.

4. Close Cites this paper Not relevant

   Mono-Forward: Revisiting Forward-Forward through Objective-Locality Decomposition

   J. Gong, Bruce Li, Waleed Abdulla · 2025 · cited 5 times · Not summarized yet

   Abstract

   Backpropagation remains the dominant algorithm for training deep neural networks, but it incurs substantial memory overhead and relies on global error propagation, which is often regarded as biologically implausible. The Forward-Forward (FF) algorithm is an appealing local-learning alternative to backpropagation, yet it still lags behind backpropagation in accuracy. A central unresolved question is whether this gap arises from FF's locality or from the positive-negative double-pass goodness objective used to train each layer. In this work, we revisit FF under the supervised setting through a decomposition that separates these two design choices. Our analysis suggests that FF's performance li…

5. Close Cites this paper Not relevant

   HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm

   Jie Yao, Hong-En Chen, C.-C. Jay Kuo · 2026 · Computer Vision and Pattern Recognition · cited 2 times · Not summarized yet

   Abstract

   Deep neural networks trained with backpropagation have achieved outstanding performance in vision tasks but remain biologically implausible, computationally demanding, and difficult to interpret. The Forward-Forward (FF) algorithm offers a promising alternative by training each layer independently through local goodness objectives. However, its purely local optimization lacks hierarchical coordination across layers, and the decoupling of goodness from features leaves the representations unconstrained and semantically ambiguous. We propose a Hierarchical and Contrastive Learning FF framework (HCL-FF) to address these limitations. HCL-FF introduces (1) a coarse-to-fine hierarchical learning st…

6. Close Not relevant

   NoProp: Training Neural Networks without Back-propagation or Forward-propagation

   Qinyu Li, Yee Whye Teh, Razvan Pascanu · 2025 · Machine Learning · cited 11 times · AI summary available

   Abstract

   The canonical deep learning approach for learning requires computing a gradient term at each layer by back-propagating the error signal from the output towards each learnable parameter. Given the stacked structure of neural networks, where each layer builds on the representation of the layer below, this approach leads to hierarchical representations. More abstract features live on the top layers of the model, while features on lower layers are expected to be less abstract. In contrast to this, we introduce a new learning method named NoProp, which does not rely on either forward or backwards propagation. Instead, NoProp takes inspiration from diffusion and flow matching methods, where each l…

7. Close Cited by this paper Not relevant

   The Predictive Forward-Forward Algorithm

   Alexander G. Ororbia, Ankur A. Mali · 2023 · Annual Meeting of the Cognitive Science Society · cited 55 times · Not summarized yet

   Abstract

   We propose the predictive forward-forward (PFF) algorithm for conducting credit assignment in neural systems. Specifically, we design a novel, dynamic recurrent neural system that learns a directed generative circuit jointly and simultaneously with a representation circuit. Notably, the system integrates learnable lateral competition, noise injection, and elements of predictive coding, an emerging and viable neurobiological process theory of cortical function, with the forward-forward (FF) adaptation scheme. Furthermore, PFF efficiently learns to propagate learning signals and updates synapses with forward passes only, eliminating key structural and computational constraints imposed by backp…

8. Close Cited by this paper Used in depth by this paper A method it uses Not relevant

   The Forward-Forward Algorithm: Some Preliminary Investigations

   Geoffrey E. Hinton · 2022 · arXiv.org · preprint, not peer reviewed · cited 498 times · Not summarized yet

   Abstract

   The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the square…

Prior work it builds on

1. Related Not relevant

   Biologically Motivated Algorithms for Propagating Local Target Representations

   Alexander G. Ororbia, Ankur A. Mali · 2018 · AAAI Conference on Artificial Intelligence · cited 108 times · Not summarized yet

   Abstract

   Finding biologically plausible alternatives to back-propagation of errors is a fundamentally important challenge in artificial neural network research. In this paper, we propose a learning algorithm called error-driven Local Representation Alignment (LRA-E), which has strong connections to predictive coding, a theory that offers a mechanistic way of describing neurocomputational machinery. In addition, we propose an improved variant of Difference Target Propagation, another procedure that comes from the same family of algorithms as LRA-E. We compare our procedures to several other biologicallymotivated algorithms, including two feedback alignment algorithms and Equilibrium Propagation. In tw…

2. Related* Not relevant

   Overfitting and neural networks: conjugate gradient and backpropagation

   S. Lawrence, C. Lee Giles · 2000 · Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New C · cited 321 times · Not summarized yet
3. Related Not relevant

   Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

   S. Ioffe, C. Szegedy · 2015 · International Conference on Machine Learning · cited 48,613 times · Not summarized yet

   Abstract

   Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful…

4. Related Used in depth by this paper A method it uses Not relevant

   Matching Networks for One Shot Learning

   O. Vinyals, C. Blundell, T. Lillicrap et al. · 2016 · Neural Information Processing Systems · cited 8,864 times · Not summarized yet

   Abstract

   Learning from a few examples remains a key challenge in machine learning. Despite recent advances in important domains such as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data. In this work, we employ ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories. Our framework learns a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. We then define one-shot learning problems on vision (using Omniglot, ImageNet) an…

5. Related Not relevant

   The neural coding framework for learning generative models

   Alexander G. Ororbia, Daniel Kifer · 2020 · Nature Communications · cited 86 times · Not summarized yet

   Abstract

   Neural generative models can be used to learn complex probability distributions from data, to sample from them, and to produce probability density estimates. We propose a computational framework for developing neural generative models inspired by the theory of predictive processing in the brain. According to predictive processing theory, the neurons in the brain form a hierarchy in which neurons in one level form expectations about sensory inputs from another level. These neurons update their local models based on differences between their expectations and the observed signals. In a similar way, artificial neurons in our generative models predict what neighboring neurons will do, and adjust …

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Forward Learning of Graph Neural Networks

   Namyong Park, Xing Wang, Antoine Simoulin et al. · 2024 · International Conference on Learning Representations · cited 4 times · Not summarized yet

   Abstract

   Graph neural networks (GNNs) have achieved remarkable success across a wide range of applications, such as recommendation, drug discovery, and question answering. Behind the success of GNNs lies the backpropagation (BP) algorithm, which is the de facto standard for training deep neural networks (NNs). However, despite its effectiveness, BP imposes several constraints, which are not only biologically implausible, but also limit the scalability, parallelism, and flexibility in learning NNs. Examples of such constraints include storage of neural activities computed in the forward pass for use in the subsequent backward pass, and the dependence of parameter updates on non-local signals. To addre…

2. Close Uses this paper in depth Not relevant

   Stochastic Forward-Forward Learning through Representational Dimensionality Compression

   Zhichao Zhu, Yang Qi, Hengyuan Ma et al. · 2025 · Neural Information Processing Systems · cited 5 times · Not summarized yet

   Abstract

   The Forward-Forward (FF) learning algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise"goodness"function with well-designed negative samples for contrastive learning. Existing goodness functions are typically defined as the sum of squared postsynaptic activations, neglecting correlated variability between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for noisy copies of individual inputs while maximizing it across the sample d…

3. Close Not relevant

   Reformulating the Forward-Forward Algorithm with Layer-wise Similarity-based Objectives

   J. Gong, Raymond Luo, Emma Wang et al. · 2026 · IEEE International Joint Conference on Neural Network · Not summarized yet

   Abstract

   Backpropagation underpins the success of artificial neural networks, yet it has critical limitations such as backward locking and global error propagation. The Forward-Forward algorithm has been proposed as a more biologically plausible method that replaces the backward pass with an additional forward pass. Nevertheless, the Forward-Forward algorithm significantly trails backpropagation in accuracy, and its optimal form requires multiple forward passes at inference time, resulting in low inference efficiency. In this work, the Forward-Forward algorithm is reformulated within a similarity-learning paradigm. The proposed algorithm is named Forward-forward Algorithm Unified with Similarity-base…

4. Close Not relevant

   Block-local learning with probabilistic latent representations

   D. Kappel, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 8 times · Not summarized yet

   Abstract

   The ubiquitous backpropagation algorithm requires sequential updates through the network introducing a locking problem. In addition, back-propagation relies on the transpose of forward weight matrices to compute updates, introducing a weight transport problem across the network. Locking and weight transport are problems because they prevent efficient parallelization and horizontal scaling of the training process. We propose a new method to address both these problems and scale up the training of large models. Our method works by dividing a deep neural network into blocks and introduces a feedback network that propagates the information from the targets backwards to provide auxiliary local lo…

5. Close Not relevant

   Energy-Efficient Supervised Learning With a Binary Stochastic Forward-Forward Algorithm

   Risi Jaiswal, Supriyo Datta, Joseph G. Makin · 2025 · IEEE Transactions on Artificial Intelligence · cited 1 time · Not summarized yet

   Abstract

   Reducing energy consumption has become a pressing need for modern machine learning, which has achieved many of its most impressive results by scaling to larger and more energy-consumptive neural networks. Unfortunately, the main algorithm for training such networks, backpropagation, poses significant challenges for custom hardware accelerators, due to both its serial dependencies and the memory footprint needed to store forward activations for the backward pass. Alternatives to backprop, although less effective, do exist; here the main computational bottleneck becomes matrix multiplication. In this article, we derive forward-forward (FF) algorithms for binary, stochastic units. Binarization …