# Towards a Rigorous Evaluation of Time-series Anomaly Detection

**Authors:** Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, Sungroh Yoon  
**arXiv:** [2109.05257](https://arxiv.org/abs/2109.05257)  
**Date:** 2021-09-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.05257v2/

## Key points

- Surge in proposed studies on time-series anomaly detection (TAD)
- High F1 scores reported on benchmark TAD datasets
- Peculiar evaluation protocol called point adjustment (PA) used
- PA has a high possibility of overestimating detection performance
- Random anomaly score can be transformed into state-of-the-art TAD method with PA
- Validity of rankings obtained through comparison of TAD methods after applying PA is questioned
- Untrained model achieves comparable detection performance to existing methods even without PA
- Current TAD methods may not be as effective as claimed
- Need for a more rigorous evaluation approach in TAD
- Proposal of new baseline and evaluation protocol for TAD to improve assessment of performance
- Background information on types of anomalies in time-series signals and their relevance to TAD datasets
- Pitfalls in evaluating TAD methods highlighted
- Experimental results supporting claims about overestimation of detection performance under PA
- Challenges prevailing evaluation practices in time-series anomaly detection
- Offers valuable insights for researchers aiming to improve upon existing methods
- Potential to enhance accuracy and reliability of future studies in this area.

## Summary

In recent years, there has been a surge in proposed studies on time-series anomaly detection (TAD) that report high F1 scores on benchmark TAD datasets, suggesting significant improvements in TAD. However, these studies often apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, the authors critically examine the PA protocol and reveal that it has a high possibility of overestimating the detection performance of TAD methods. They demonstrate that even a random anomaly score can easily be transformed into a state-of-the-art TAD method when PA is applied. This raises concerns about the validity of rankings obtained through the comparison of TAD methods after applying PA. Furthermore, the authors question the potential of existing TAD methods by showing that an untrained model achieves comparable detection performance to existing methods even when PA is forbidden. These findings suggest that current TAD methods may not be as effective as claimed and highlight the need for a more rigorous evaluation approach. Based on their insights, the authors propose a new baseline and evaluation protocol for TAD to facilitate a more rigorous assessment of its performance. By addressing the limitations of the existing evaluation practices, they aim to improve future research in this field. The paper provides background information on different types of anomalies in time-series signals and discusses their relevance to TAD datasets. It also highlights some pitfalls in evaluating TAD methods and presents experimental results to support their claims about the overestimation of detection performance under PA. Overall, this study challenges prevailing evaluation practices in time-series anomaly detection and offers valuable insights for researchers aiming to improve upon existing methods. The proposed baseline and evaluation protocol have the potential to enhance accuracy and reliability of future studies in this area.