# Efficient Off-Policy Q-Learning for Data-Based Discrete-Time LQR Problems

**Authors:** Victor G. Lopez, Mohammad Alsalti, Matthias A. Müller  
**arXiv:** [2105.07761](https://arxiv.org/abs/2105.07761)  
**Date:** 2021-05-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2105.07761v3/

## Key points

- Presents an improved Q-learning algorithm for discrete-time linear time-invariant systems
- Does not require any knowledge of the system dynamics
- Offers significant efficiency advantages over other data-based optimal control methods in the literature
- Can be fully executed off-line without applying the current estimate of the optimal input to the system as in on-policy algorithms
- Convergence is guaranteed by a persistently exciting input defined from an easily tested matrix rank condition
- A data-based method is proposed to design the initial stabilizing feedback gain that the algorithm requires
- Robustness of the algorithm in the presence of noisy measurements is analyzed
- Compared in simulation to different direct and indirect data-based control design methods, outperforms existing techniques
- Provides a promising solution for solving discrete-time LQR problems
- Improves data-based optimal control methods' efficiency and effectiveness while requiring minimal prior knowledge about system dynamics

## Summary

This paper presents an improved Q-learning algorithm for discrete-time linear time-invariant systems that does not require any knowledge of the system dynamics. The proposed method offers significant efficiency advantages over other data-based optimal control methods in the literature and can be fully executed off-line without applying the current estimate of the optimal input to the system as in on-policy algorithms. The algorithm's convergence is guaranteed by a persistently exciting input defined from an easily tested matrix rank condition. Additionally, a data-based method is proposed to design the initial stabilizing feedback gain that the algorithm requires. The authors analyze the robustness of the algorithm in the presence of noisy measurements. To evaluate its performance, the proposed algorithm is compared in simulation to different direct and indirect data-based control design methods. The results demonstrate that this new approach outperforms existing techniques and provides a promising solution for solving discrete-time LQR problems. Overall, this paper contributes significantly to improving data-based optimal control methods' efficiency and effectiveness while requiring minimal prior knowledge about system dynamics. It has practical implications for various applications where accurate models are difficult or expensive to obtain but require optimal control solutions.