The Cascaded Forward Algorithm for Neural Network Training

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Cite cet article Pas pertinent

   Training convolutional neural networks with the Forward–Forward Algorithm

   Riccardo Scodellaro, A. Kulkarni, Frauke Alves et al. · 2023 · Scientific Reports · cité 27 fois · Pas encore résumé

   Résumé scientifique

   Recent successes in image analysis with deep neural networks are achieved almost exclusively with Convolutional Neural Networks (CNNs), typically trained using the backpropagation (BP) algorithm. In a 2022 preprint, Geoffrey Hinton proposed the Forward–Forward (FF) algorithm as a biologically inspired alternative, where positive and negative examples are jointly presented to the network and training is guided by a locally defined goodness function. Here, we extend the FF paradigm to CNNs. We introduce two spatially extended labeling strategies, based on Fourier patterns and morphological transformations, that enable convolutional layers to access label information across all spatial position…

2. Très proche Pas pertinent

   Scalable Forward-Forward Algorithm

   Andrii Krutsylo · 2025 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

   Résumé scientifique

   We propose a scalable Forward-Forward (FF) algorithm that eliminates the need for backpropagation by training each layer separately. Unlike backpropagation, FF avoids backward gradients and can be more modular and memory efficient, making it appealing for large networks. We extend FF to modern convolutional architectures, such as MobileNetV3 and ResNet18, by introducing a new way to compute losses for convolutional layers. Experiments show that our method achieves performance comparable to standard backpropagation. Furthermore, when we divide the network into blocks, such as the residual blocks in ResNet, and apply backpropagation only within each block, but not across blocks, our hybrid des…

3. Très proche Cite cet article Pas pertinent

   Exploring the Feasibility of Forward Forward Algorithm in Neural Networks

   Suraj R Gautham, S. Nair, Suresh Jamadagni et al. · 2024 · 2024 International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE) · cité 1 fois · Pas encore résumé

   Résumé scientifique

   This research explores the Forward-Forward Algorithm, proposed as an alternative to Backpropagation, in the context of Convolutional and Recurrent Neural Networks for multiclass classification. The authors' investigation aims to evaluate the algorithm's efficiency, advantages, and disadvantages for real-world applications. Results offer insights into the untapped potential and nuances of this approach, guiding researchers in optimizing neural network performance and potentially advancing a more human-like approach in Deep Learning.

4. Proche Cite cet article Pas pertinent

   Mono-Forward: Revisiting Forward-Forward through Objective-Locality Decomposition

   J. Gong, Bruce Li, Waleed Abdulla · 2025 · cité 5 fois · Pas encore résumé

   Résumé scientifique

   Backpropagation remains the dominant algorithm for training deep neural networks, but it incurs substantial memory overhead and relies on global error propagation, which is often regarded as biologically implausible. The Forward-Forward (FF) algorithm is an appealing local-learning alternative to backpropagation, yet it still lags behind backpropagation in accuracy. A central unresolved question is whether this gap arises from FF's locality or from the positive-negative double-pass goodness objective used to train each layer. In this work, we revisit FF under the supervised setting through a decomposition that separates these two design choices. Our analysis suggests that FF's performance li…

5. Proche Cite cet article Pas pertinent

   HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm

   Jie Yao, Hong-En Chen, C.-C. Jay Kuo · 2026 · Computer Vision and Pattern Recognition · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Deep neural networks trained with backpropagation have achieved outstanding performance in vision tasks but remain biologically implausible, computationally demanding, and difficult to interpret. The Forward-Forward (FF) algorithm offers a promising alternative by training each layer independently through local goodness objectives. However, its purely local optimization lacks hierarchical coordination across layers, and the decoupling of goodness from features leaves the representations unconstrained and semantically ambiguous. We propose a Hierarchical and Contrastive Learning FF framework (HCL-FF) to address these limitations. HCL-FF introduces (1) a coarse-to-fine hierarchical learning st…

6. Proche Pas pertinent

   NoProp: Training Neural Networks without Back-propagation or Forward-propagation

   Qinyu Li, Yee Whye Teh, Razvan Pascanu · 2025 · Machine Learning · cité 11 fois · Résumé IA disponible

   Résumé scientifique

   The canonical deep learning approach for learning requires computing a gradient term at each layer by back-propagating the error signal from the output towards each learnable parameter. Given the stacked structure of neural networks, where each layer builds on the representation of the layer below, this approach leads to hierarchical representations. More abstract features live on the top layers of the model, while features on lower layers are expected to be less abstract. In contrast to this, we introduce a new learning method named NoProp, which does not rely on either forward or backwards propagation. Instead, NoProp takes inspiration from diffusion and flow matching methods, where each l…

7. Proche Cité par cet article Pas pertinent

   The Predictive Forward-Forward Algorithm

   Alexander G. Ororbia, Ankur A. Mali · 2023 · Annual Meeting of the Cognitive Science Society · cité 55 fois · Pas encore résumé

   Résumé scientifique

   We propose the predictive forward-forward (PFF) algorithm for conducting credit assignment in neural systems. Specifically, we design a novel, dynamic recurrent neural system that learns a directed generative circuit jointly and simultaneously with a representation circuit. Notably, the system integrates learnable lateral competition, noise injection, and elements of predictive coding, an emerging and viable neurobiological process theory of cortical function, with the forward-forward (FF) adaptation scheme. Furthermore, PFF efficiently learns to propagate learning signals and updates synapses with forward passes only, eliminating key structural and computational constraints imposed by backp…

8. Proche Cité par cet article Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   The Forward-Forward Algorithm: Some Preliminary Investigations

   Geoffrey E. Hinton · 2022 · arXiv.org · preprint, non relu par les pairs · cité 498 fois · Pas encore résumé

   Résumé scientifique

   The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth further investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the square…

Les travaux antérieurs sur lesquels il s’appuie

1. Apparenté Pas pertinent

   Biologically Motivated Algorithms for Propagating Local Target Representations

   Alexander G. Ororbia, Ankur A. Mali · 2018 · AAAI Conference on Artificial Intelligence · cité 108 fois · Pas encore résumé

   Résumé scientifique

   Finding biologically plausible alternatives to back-propagation of errors is a fundamentally important challenge in artificial neural network research. In this paper, we propose a learning algorithm called error-driven Local Representation Alignment (LRA-E), which has strong connections to predictive coding, a theory that offers a mechanistic way of describing neurocomputational machinery. In addition, we propose an improved variant of Difference Target Propagation, another procedure that comes from the same family of algorithms as LRA-E. We compare our procedures to several other biologicallymotivated algorithms, including two feedback alignment algorithms and Equilibrium Propagation. In tw…

2. Apparenté* Pas pertinent

   Overfitting and neural networks: conjugate gradient and backpropagation

   S. Lawrence, C. Lee Giles · 2000 · Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New C · cité 321 fois · Pas encore résumé
3. Apparenté Pas pertinent

   Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

   S. Ioffe, C. Szegedy · 2015 · International Conference on Machine Learning · cité 48 613 fois · Pas encore résumé

   Résumé scientifique

   Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful…

4. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Matching Networks for One Shot Learning

   O. Vinyals, C. Blundell, T. Lillicrap et al. · 2016 · Neural Information Processing Systems · cité 8 864 fois · Pas encore résumé

   Résumé scientifique

   Learning from a few examples remains a key challenge in machine learning. Despite recent advances in important domains such as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data. In this work, we employ ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories. Our framework learns a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. We then define one-shot learning problems on vision (using Omniglot, ImageNet) an…

5. Apparenté Pas pertinent

   The neural coding framework for learning generative models

   Alexander G. Ororbia, Daniel Kifer · 2020 · Nature Communications · cité 86 fois · Pas encore résumé

   Résumé scientifique

   Neural generative models can be used to learn complex probability distributions from data, to sample from them, and to produce probability density estimates. We propose a computational framework for developing neural generative models inspired by the theory of predictive processing in the brain. According to predictive processing theory, the neurons in the brain form a hierarchy in which neurons in one level form expectations about sensory inputs from another level. These neurons update their local models based on differences between their expectations and the observed signals. In a similar way, artificial neurons in our generative models predict what neighboring neurons will do, and adjust …

Les articles qui le citent

1. Proche Utilise cet article en profondeur Pas pertinent

   Forward Learning of Graph Neural Networks

   Namyong Park, Xing Wang, Antoine Simoulin et al. · 2024 · International Conference on Learning Representations · cité 4 fois · Pas encore résumé

   Résumé scientifique

   Graph neural networks (GNNs) have achieved remarkable success across a wide range of applications, such as recommendation, drug discovery, and question answering. Behind the success of GNNs lies the backpropagation (BP) algorithm, which is the de facto standard for training deep neural networks (NNs). However, despite its effectiveness, BP imposes several constraints, which are not only biologically implausible, but also limit the scalability, parallelism, and flexibility in learning NNs. Examples of such constraints include storage of neural activities computed in the forward pass for use in the subsequent backward pass, and the dependence of parameter updates on non-local signals. To addre…

2. Proche Utilise cet article en profondeur Pas pertinent

   Stochastic Forward-Forward Learning through Representational Dimensionality Compression

   Zhichao Zhu, Yang Qi, Hengyuan Ma et al. · 2025 · Neural Information Processing Systems · cité 5 fois · Pas encore résumé

   Résumé scientifique

   The Forward-Forward (FF) learning algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise"goodness"function with well-designed negative samples for contrastive learning. Existing goodness functions are typically defined as the sum of squared postsynaptic activations, neglecting correlated variability between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for noisy copies of individual inputs while maximizing it across the sample d…

3. Proche Pas pertinent

   Reformulating the Forward-Forward Algorithm with Layer-wise Similarity-based Objectives

   J. Gong, Raymond Luo, Emma Wang et al. · 2026 · IEEE International Joint Conference on Neural Network · Pas encore résumé

   Résumé scientifique

   Backpropagation underpins the success of artificial neural networks, yet it has critical limitations such as backward locking and global error propagation. The Forward-Forward algorithm has been proposed as a more biologically plausible method that replaces the backward pass with an additional forward pass. Nevertheless, the Forward-Forward algorithm significantly trails backpropagation in accuracy, and its optimal form requires multiple forward passes at inference time, resulting in low inference efficiency. In this work, the Forward-Forward algorithm is reformulated within a similarity-learning paradigm. The proposed algorithm is named Forward-forward Algorithm Unified with Similarity-base…

4. Proche Pas pertinent

   Block-local learning with probabilistic latent representations

   D. Kappel, Khaleelulla Khan Nazeer, Cabrel Teguemne Fokam et al. · 2023 · arXiv.org · preprint, non relu par les pairs · cité 8 fois · Pas encore résumé

   Résumé scientifique

   The ubiquitous backpropagation algorithm requires sequential updates through the network introducing a locking problem. In addition, back-propagation relies on the transpose of forward weight matrices to compute updates, introducing a weight transport problem across the network. Locking and weight transport are problems because they prevent efficient parallelization and horizontal scaling of the training process. We propose a new method to address both these problems and scale up the training of large models. Our method works by dividing a deep neural network into blocks and introduces a feedback network that propagates the information from the targets backwards to provide auxiliary local lo…

5. Proche Pas pertinent

   Energy-Efficient Supervised Learning With a Binary Stochastic Forward-Forward Algorithm

   Risi Jaiswal, Supriyo Datta, Joseph G. Makin · 2025 · IEEE Transactions on Artificial Intelligence · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Reducing energy consumption has become a pressing need for modern machine learning, which has achieved many of its most impressive results by scaling to larger and more energy-consumptive neural networks. Unfortunately, the main algorithm for training such networks, backpropagation, poses significant challenges for custom hardware accelerators, due to both its serial dependencies and the memory footprint needed to store forward activations for the backward pass. Alternatives to backprop, although less effective, do exist; here the main computational bottleneck becomes matrix multiplication. In this article, we derive forward-forward (FF) algorithms for binary, stochastic units. Binarization …