# ReLU-KAN: New Kolmogorov-Arnold Networks that Only Need Matrix Addition, Dot Multiplication, and ReLU

**Authors:** Qi Qiu, Tao Zhu, Helin Gong, Liming Chen, Huansheng Ning  
**arXiv:** [2406.02075](https://arxiv.org/abs/2406.02075)  
**Date:** 2024-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.02075v1/

## Key points

- Kolmogorov-Arnold Networks (KAN) face challenges due to complex B-spline computations, limiting parallel computing on GPUs
- Researchers Qi Qiu, Tao Zhu, Helin Gong, Liming Chen, and Huansheng Ning introduce ReLU-KAN as a solution
- ReLU-KAN simplifies KAN's basis function design using ReLU activation functions and point-wise multiplication for efficient CUDA computing
- ReLU-KAN is compatible with PyTorch and outperforms traditional KAN models with 4-layer networks by achieving a 20x speedup
- The architecture exhibits enhanced stability in training, superior fitting capabilities, and retains "catastrophic forgetting avoidance"
- ReLU-KAN leverages matrix addition, dot multiplication, and ReLU activation functions for improved efficiency in deep learning applications

## Summary

In the realm of deep learning, Kolmogorov-Arnold Networks (KAN) have been hindered by the intricate calculations involved in basis function (B-spline) computations. This has led to limitations in parallel computing capabilities on GPUs. To address this issue, a group of researchers including Qi Qiu, Tao Zhu, Helin Gong, Liming Chen, and Huansheng Ning have introduced a groundbreaking solution known as ReLU-KAN. This innovative implementation stays true to the fundamental principles of KAN while incorporating Rectified Linear Unit (ReLU) activation functions and point-wise multiplication techniques. By doing so, the design of KAN's basis function is simplified and the computation process is optimized for efficient CUDA computing. One of the key advantages of the proposed ReLU-KAN architecture is its compatibility with existing deep learning frameworks such as PyTorch. This enables seamless implementation for both inference and training tasks. Through rigorous experimentation, it has been demonstrated that ReLU-KAN outperforms traditional KAN models with 4-layer networks by achieving an impressive 20x speedup. Moreover, ReLU-KAN exhibits enhanced stability during the training process and showcases superior fitting capabilities while retaining the crucial "catastrophic forgetting avoidance" property characteristic of KAN. The research conducted by Qi Qiu et al. culminates in a significant contribution to the field of deep learning by introducing a more efficient and effective neural network architecture that leverages matrix addition, dot multiplication, and ReLU activation functions. For those interested in exploring further or implementing this cutting-edge technology themselves, the code for ReLU-KAN can be accessed via https://github.com/quiqi/relu_kan.