Dynamic ReLU

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Not relevant

   DPReLU: Dynamic Parametric Rectified Linear Unit

   Kien Mai Ngoc, Dong-Hun Yang, Iksoo Shin et al. · 2020 · International Conference on Smart Media and Applications · cited 5 times · Not summarized yet

   Abstract

   As one of the essential elements of deep learning, the rectified linear unit (ReLU) is the most widely used activation function to solve the vanishing gradient problem. However, because of the dying ReLU problem and bias shift effect, deep learning models with ReLU may miss the potential benefits of negative values. To alleviate this problem, numerous ReLU variants have been proposed. In this paper, we propose Dynamic Parametric ReLU (DPReLU) that can control the overall functional shape of ReLU with four learnable parameters and is based on the ideas of the Parametric ReLU (PReLU) and Flexible ReLU (FReLU). The parameters of DPReLU are determined by training rather than by humans, thereby m…

2. Very close Not relevant

   DPReLU: Dynamic Parametric Rectified Linear Unit and Its Proper Weight Initialization Method

   Dong-Hun Yang, Kien Mai Ngoc, Iksoo Shin et al. · 2023 · International Journal of Computational Intelligence Systems · cited 20 times · Not summarized yet

   Abstract

   Activation functions are essential in deep learning, and the rectified linear unit (ReLU) is the most widely used activation function to solve the vanishing gradient problem. However, owing to the dying ReLU problem and bias shift effect, deep learning models using ReLU cannot exploit the potential benefits of negative values. Numerous ReLU variants have been proposed to address this issue. In this study, we propose Dynamic Parametric ReLU (DPReLU), which can dynamically control the overall functional shape of ReLU with four learnable parameters. The parameters of DPReLU are determined by training rather than by humans, thereby making the formulation more suitable and flexible for each model…

3. Very close Not relevant

   Flexible Rectified Linear Units for Improving Convolutional Neural Networks

   Suo Qiu, Bo-Lun Cai · 2017 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet

   Abstract

   Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. In this paper, we propose a novel activation function called flexible rectified linear unit (FReLU). FReLU improves the flexibility of ReLU by a learnable rectified point. FReLU achieves a faster convergence and higher performance. Furthermore, FReLU does not rely on strict assumptions by self-adaption. FReLU is also simple and effective without using exponential function. We evaluate FReLU on two standard image classification dataset, including CIFAR-10 and CIFAR-100. Experimental results show the strengths of the proposed method.

4. Very close Not relevant

   FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

   Suo Qiu, Xiang-Min Xu, Bo-Lun Cai · 2018 · International Conference on Pattern Recognition · cited 96 times · Not summarized yet

   Abstract

   Rectified linear unit (ReLU) is a widely used activation function for deep convolutional neural networks. However, because of the zero-hard rectification, ReLU networks lose the benefits from negative values. In this paper, we propose a novel activation function called flexible rectified linear unit (FReLU) to further explore the effects of negative values. By redesigning the rectified point of ReLU as a learnable parameter, FReLU expands the states of the activation output. When a network is successfully trained, FReLU tends to converge to a negative value, which improves the expressiveness and thus the performance. Furthermore, FReLU is designed to be simple and effective without exponenti…

5. Very close Not relevant

   Adaptive Convolutional ReLUs

   Hongyang Gao, Lei Cai, Shuiwang Ji · 2020 · AAAI Conference on Artificial Intelligence · cited 17 times · Not summarized yet

   Abstract

   Rectified linear units (ReLUs) are currently the most popular activation function used in neural networks. Although ReLUs can solve the gradient vanishing problem and accelerate training convergence, it suffers from the dying ReLU problem in which some neurons are never activated if the weights are not updated properly. In this work, we propose a novel activation function, known as the adaptive convolutional ReLU (ConvReLU), that can better mimic brain neuron activation behaviors and overcome the dying ReLU problem. With our novel parameter sharing scheme, ConvReLUs can be applied to convolution layers that allow each input neuron to be activated by different trainable thresholds without inv…

6. Very close* Not relevant

   Deep Learning using Rectified Linear Units (ReLU)

   Abien Fred Agarap · 2018 · arXiv.org · preprint, not peer reviewed · cited 4,021 times · Not summarized yet
7. Very close Not relevant

   Deep neural networks with Elastic Rectified Linear Units for object recognition

   Xiao-Heng Jiang, Yan-Wei Pang, Xue-Long Li et al. · 2018 · Neurocomputing · cited 129 times · Not summarized yet

   Abstract

   Abstract Rectified Linear Unit (ReLU) is crucial to the recent success of deep neural networks (DNNs). In this paper, we propose a novel Elastic Rectified Linear Unit (EReLU) that focuses on processing the positive part of input. Unlike previous variants of ReLU that typically adopt linear or piecewise linear functions to represent the positive part, EReLU is characterized by that each positive value scales within a moderate range like a spring during training stage. On test time, EReLU becomes standard ReLU. EReLU improves model fitting with no extra parameters and little overfitting risk. Furthermore, we propose Elastic Parametric Rectified Linear Unit (EPReLU) by taking advantage of EReLU…

8. Very close Not relevant

   A ReLU Dense Layer to Improve the Performance of Neural Networks

   Alireza M. Javid, Sandipan Das, M. Skoglund et al. · 2020 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cited 45 times · Not summarized yet

   Abstract

   We propose ReDense as a simple and low complexity way to improve the performance of trained neural networks. We use a combination of random weights and rectified linear unit (ReLU) activation function to add a ReLU dense (ReDense) layer to the trained neural network such that it can achieve a lower training loss. The lossless flow property (LFP) of ReLU is the key to achieve the lower training loss while keeping the generalization error small. ReDense does not suffer from vanishing gradient problem in the training due to having a shallow structure. We experimentally show that ReDense can improve the training and testing performance of various neural network architectures with different optim…

Prior work it builds on

1. Close Not relevant

   Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

   Djork-Arné Clevert, T. Unterthiner, S. Hochreiter · 2015 · International Conference on Learning Representations · cited 6,493 times · Not summarized yet

   Abstract

   We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit na…

2. Close Not relevant

   Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

   Ludovic Trottier, Philippe Gigu�re, B. Chaib-draa · 2016 · International Conference on Machine Learning and Applications · cited 226 times · Not summarized yet

   Abstract

   Object recognition is an important task for improving the ability of visual systems to perform complex scene understanding. Recently, the Exponential Linear Unit (ELU) has been proposed as a key component for managing bias shift in Convolutional Neural Networks (CNNs), but defines a parameter that must be set by hand. In this paper, we propose learning a parameterization of ELU in order to learn the proper activation shape at each layer in the CNNs. Our results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN, Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU (PELU) has better performances than the non-parametric ELU. We have observed as much as a …

3. Close Not relevant

   Swish: a Self-Gated Activation Function

   Prajit Ramachandran, Barret Zoph, Quoc V. Le · 2017 · arXiv: Neural and Evolutionary Computing · preprint, not peer reviewed · cited 728 times · Not summarized yet

   Abstract

   The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose a new activation function, named Swish, which is simply $f(x) = x \cdot \text{sigmoid}(x)$. Our experiments show that Swish tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for M…

4. Related A method it uses Not relevant

   BlockDrop: Dynamic Inference Paths in Residual Networks

   Zu-Xuan Wu, Tushar Nagarajan, Abhishek Kumar et al. · 2017 · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · cited 563 times · Not summarized yet

   Abstract

   Very deep convolutional neural networks offer excellent recognition results, yet their computational expense limits their impact for many real-world applications. We introduce BlockDrop, an approach that learns to dynamically choose which layers of a deep network to execute during inference so as to best reduce total computation without degrading prediction accuracy. Exploiting the robustness of Residual Networks (ResNets) to layer dropping, our framework selects on-the-fly which residual blocks to evaluate for a given novel image. In particular, given a pretrained ResNet, we train a policy network in an associative reinforcement learning setting for the dual reward of utilizing a minimal nu…

5. Related Used in depth by this paper A method it uses Not relevant

   Dynamic Convolution: Attention Over Convolution Kernels

   Yinpeng Chen, Xi-Yang Dai, Meng-Chen Liu et al. · 2019 · Computer Vision and Pattern Recognition · cited 1,542 times · Not summarized yet

   Abstract

   Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability. To address this issue, we present Dynamic Convolution, a new design that increases model complexity without increasing the network depth or width. Instead of using a single convolution kernel per layer, dynamic convolution aggregates multiple parallel convolution kernels dynamically based upon their attentions, which are input dependent. Assembling multiple kernels is not only computationally efficient due to the small kernel size, …

Papers that cite it

1. Close Uses this paper in depth Not relevant

   PWLU: Learning Specialized Activation Functions With the Piecewise Linear Unit

   Zezhou Zhu, Yu-Cong Zhou, Yuan Dong et al. · 2023 · IEEE Transactions on Pattern Analysis and Machine Intelligence · cited 15 times · Not summarized yet

   Abstract

   The choice of activation functions is crucial to deep neural networks. ReLU is a popular hand-designed activation function. Swish, the automatically searched activation function, outperforms ReLU on many challenging datasets. However, the search method has two main drawbacks. First, the tree-based search space is highly discrete and restricted, which is difficult to search. Second, the sample-based search method is inefficient in finding specialized activation functions for each dataset or neural architecture. To overcome these drawbacks, we propose a new activation function called Piecewise Linear Unit (PWLU), incorporating a carefully designed formulation and learning method. PWLU can lear…

2. Close Uses this paper in depth Not relevant

   RepAct: The Re-parameterizable Adaptive Activation Function

   Xian Wu, Qingchuan Tao, Shuang Wang · 2024 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Addressing the imperative need for efficient artificial intelligence in IoT and edge computing, this study presents RepAct, a re-parameterizable adaptive activation function tailored for optimizing lightweight neural networks within the computational limitations of edge devices. By employing a multi-branch structure with learnable adaptive weights, RepAct enriches feature processing and enhances cross-layer interpretability. When evaluated on tasks such as image classification and object detection, RepAct notably surpassed conventional activation functions in lightweight networks, delivering up to a 7.92% accuracy boost on MobileNetV3-Small for the ImageNet100 dataset, while maintaining comp…

3. Close Not relevant

   VeLU: Variance-enhanced Learning Unit for Deep Neural Networks

   Ashkan Shakarami, Yousef Yeganeh, Azade Farshad et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   Abstract

   Activation functions play a critical role in deep neural networks by shaping gradient flow, optimization stability, and generalization. While ReLU remains widely used due to its simplicity, it suffers from gradient sparsity and dead-neuron issues and offers no adaptivity to input statistics. Smooth alternatives such as Swish and GELU improve gradient propagation but still apply a fixed transformation regardless of the activation distribution. In this paper, we propose VeLU, a Variance-enhanced Learning Unit that introduces variance-aware and distributionally aligned nonlinearity through a principled combination of ArcTan-ArcSin transformations, adaptive scaling, and Wasserstein-2 regularizat…

4. Related Uses this paper in depth Not relevant

   MicroNet: Improving Image Recognition with Extremely Low FLOPs

   Yun-Sheng Li, Yinpeng Chen, Xi-Yang Dai et al. · 2021 · IEEE International Conference on Computer Vision · cited 114 times · Not summarized yet

   Abstract

   This paper aims at addressing the problem of substantial performance degradation at extremely low computational cost (e.g. 5M FLOPs on ImageNet classification). We found that two factors, sparse connectivity and dynamic activation function, are effective to improve the accuracy. The former avoids the significant reduction of network width, while the latter mitigates the detriment of reduction in network depth. Technically, we propose micro-factorized convolution, which factorizes a convolution matrix into low rank matrices, to integrate sparse connectivity into convolution. We also present a new dynamic activation function, named Dynamic Shift Max, to improve the non-linearity via maxing out…

5. Close Not relevant

   Web-aided data set expansion in deep learning: evaluating trainable activation functions in ResNet for improved image classification

   Zhiqiang Zhang, Xiaoming Li, Xinyi Xu et al. · 2024 · International Journal of Web Information Systems · cited 2 times · Not summarized yet

   Abstract

   Purpose The purpose of this study is to explore the potential of trainable activation functions to enhance the performance of deep neural networks, specifically ResNet architectures, in the task of image classification. By introducing activation functions that adapt during training, the authors aim to determine whether such flexibility can lead to improved learning outcomes and generalization capabilities compared to static activation functions like ReLU. This research seeks to provide insights into how dynamic nonlinearities might influence deep learning models' efficiency and accuracy in handling complex image data sets. Design/methodology/approach This research integrates three novel trai…