# MemFlow: Optical Flow Estimation and Prediction with Memory

**Authors:** Qiaole Dong, Yanwei Fu  
**arXiv:** [2404.04808](https://arxiv.org/abs/2404.04808)  
**Date:** 2024-04-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.04808v1/

## Key points

- Groundbreaking method for optical flow estimation and prediction with memory
- Real-time solution leveraging memory read-out and update modules
- Effective historical motion aggregation enhances temporal coherence
- Resolution-adaptive re-scaling accommodates diverse video resolutions effectively
- Capabilities extended to predict optical flow based on past observations
- Surpasses VideoFlow performance with fewer parameters and faster inference speed on benchmark datasets like Sintel and KITTI-15
- Leads in performance on the 1080p Spring dataset at the time of submission
- Introducing long-term memory does not significantly impact performance, opening avenues for future research into exploring long-range motion history for optical flow estimation while maintaining efficiency for real-time applications

## Summary

is a groundbreaking method for optical flow estimation and prediction with memory. It addresses the limitations of existing approaches in the vision community by introducing a real-time solution that leverages memory read-out and update modules. Traditional optical flow estimation techniques rely on two frames as input, while newer methods incorporate multiple frames to capture long-range information. However, these methods struggle to fully exploit temporal coherence or suffer from high computational overhead, making real-time flow estimation challenging. Through effective historical motion aggregation,  not only enhances temporal coherence but also enables resolution-adaptive re-scaling to accommodate diverse video resolutions effectively. Additionally, it extends its capabilities to predict optical flow based on past observations, offering a comprehensive solution for dynamic environments. This innovative approach surpasses the performance of VideoFlow with fewer parameters and faster inference speed on benchmark datasets like Sintel and KITTI-15 in terms of generalization performance. At the time of submission,  leads in performance on the 1080p Spring dataset, showcasing its superior predictive capabilities. Furthermore, ablation studies demonstrate that introducing long-term memory does not significantly impact performance but opens up avenues for future research into exploring long-range motion history for optical flow estimation while maintaining efficiency for real-time applications. In conclusion,  stands out as a novel online approach that revolutionizes video-based optical flow estimation by incorporating memory mechanisms and resolution-adaptive techniques for top-notch prediction performance in safety-critical scenarios.