# Accelerating Rescaled Gradient Descent: Fast Optimization of Smooth Functions

**Authors:** Ashia Wilson, Lester Mackey, Andre Wibisono  
**arXiv:** [1902.08825](https://arxiv.org/abs/1902.08825)  
**Date:** 2019-02-23  
**Summary page:** https://www.summarizepaper.com/en/paper/1902.08825v3/

## Key points

- The paper introduces a family of algorithms called descent algorithms for optimizing convex and non-convex functions.
- A new first-order algorithm called rescaled gradient descent (RGD) is proposed, which has a faster convergence rate compared to traditional gradient descent methods under the assumption of strong smoothness of the objective function.
- Two frameworks are presented for accelerating descent methods in the case of convex objective functions: one following Nesterov's style and the other adopting Monteiro and Svaiter's style, both utilizing a single Lyapunov function.
- RGD can be accelerated using these frameworks under the same strong smoothness assumption.
- Examples of strongly smooth loss functions in machine learning are provided to support the theoretical findings, along with numerical experiments validating the results.
- Extensions to universal tensor methods and coordinate settings are presented for practical optimization problems.
- The work contributes to advancing optimization techniques by introducing descent algorithms and demonstrating their effectiveness in optimizing both convex and non-convex functions with improved convergence rates when applied to strongly smooth objective functions.

## Summary

The paper introduces a family of algorithms called descent algorithms, designed for optimizing convex and non-convex functions. The authors propose a new first-order algorithm called rescaled gradient descent (RGD), which demonstrates a faster convergence rate compared to traditional gradient descent methods under the assumption that the objective function is strongly smooth. This strong smoothness assumption is an extension of the standard smoothness assumption commonly used in optimization. In the case of convex objective functions, two frameworks are presented for accelerating descent methods: one following the style of Nesterov and the other adopting Monteiro and Svaiter's style. Both utilize a single Lyapunov function. It is shown that RGD can be accelerated using these frameworks under the same strong smoothness assumption. To support their theoretical findings, examples of strongly smooth loss functions in machine learning are provided along with numerical experiments to validate their results. Additionally, extensions to universal tensor methods and coordinate settings are presented for practical optimization problems. Overall, this work contributes to advancing optimization techniques by introducing descent algorithms and demonstrating their effectiveness in optimizing both convex and non-convex functions with improved convergence rates when applied to strongly smooth objective functions.