# Liquid Time-constant Networks

**Authors:** Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, Radu Grosu  
**arXiv:** [2006.04439](https://arxiv.org/abs/2006.04439)  
**Date:** 2020-06-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2006.04439v1/

## Key points

- Introduction of Liquid Time-Constant Networks (LTCs), a new class of time-continuous recurrent neural network models
- LTCs impose specialized nonlinearities on network connections, resulting in dynamical systems with varying time-constants
- LTCs exhibit stable and bounded behavior, providing superior expressivity within the family of neural ordinary differential equations (ODEs)
- Theoretical bounds and measure of "trajectory length" used to analyze properties and expressive power of LTCs
- LTCs outperform classical and modern recurrent neural networks in time-series prediction experiments
- Potential applications of LTCs in domains such as robot control in continuous-time observation and action spaces
- Time-continuous models gaining popularity due to benefits like adaptive computations, better continuous time-series modeling, memory efficiency, and parameter efficiency
- Investigation into expressive power of neural ODEs and proposal of LTC as an enhanced ODE model
- Discussion on measures of expressivity used in previous works to understand why deeper networks perform well compared to shallow networks
- Lower bounds for trajectory length growth highlight the superior approximation capabilities of LTCs

## Summary

In this study, the authors introduce a new class of time-continuous recurrent neural network models called Liquid Time-Constant Networks (LTCs). Unlike traditional neural networks that rely on nonlinearities of individual neurons, LTCs impose specialized nonlinearities on the network connections. This results in dynamical systems with varying time-constants coupled to their hidden state and outputs computed by numerical differential equation solvers. The authors demonstrate that LTCs exhibit stable and bounded behavior, providing superior expressivity within the family of neural ordinary differential equations (ODEs). To analyze the properties of LTCs, they first establish theoretical bounds over their dynamics and compute their expressive power using a measure called "trajectory length" in a latent trajectory representation space. To validate the effectiveness of LTCs, the authors conduct a series of time-series prediction experiments. The results show that LTCs outperform classical and modern recurrent neural networks in terms of approximation capability. Looking ahead, the authors suggest exploring the causality of performant recurrent models like LTCs, as their semantics resemble dynamic causal models. They propose that LTCs could be applied in domains such as robot control in continuous-time observation and action spaces to improve reasoning. In related works, time-continuous models have gained popularity due to benefits like adaptive computations, better continuous time-series modeling, memory efficiency and parameter efficiency. Various approaches have been proposed to improve and stabilize these models. The authors contribute by investigating the expressive power of neural ODEs and proposing a new ODE model (LTC) that enhances expressivity and performance. The study also discusses measures of expressivity used in previous works to understand why deeper networks perform well compared to shallow networks. These measures include counting linear regions in neural networks; studying classes of radial functions produced by smaller networks; and analyzing exponential expressivity through transient chaos. The authors extend these analyses to time-continuous networks and provide lower bounds for trajectory length growth which highlights the superior approximation capabilities of LTCs. In conclusion, the authors demonstrate the effectiveness of LTCs - a novel class of time-continuous neural network models - showing that they can be implemented effectively using various ODE solvers and trained through backpropagation while exhibiting bounded and stable dynamics with superior expressivity than standard or modern deep learning models when used for supervised learning tasks involving time series data prediction tasks such as those found in medical applications or automation sectors..