# Foundation Models for Structural Health Monitoring

**Authors:** Luca Benfenati, Daniele Jahier Pagliari, Luca Zanatta, Yhorman Alexander Bedoya Velez, Andrea Acquaviva, Massimo Poncino, Enrico Macii, Luca Benini, Alessio Burrello  
**arXiv:** [2404.02944](https://arxiv.org/abs/2404.02944)  
**Date:** 2024-04-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.02944v1/

## Key points

- Structural Health Monitoring (SHM) is crucial for ensuring safety and reliability of civil infrastructures, especially bridges and viaducts.
- Transformer neural networks with a Masked Auto-Encoder architecture are used as Foundation Models for SHM, outperforming traditional methods in tasks like Anomaly Detection (AD) and Traffic Load Estimation (TLE).
- The innovation of Transformer architecture lies in self-attention mechanisms that efficiently capture long-range dependencies.
- Masked Autoencoders serve as scalable self-supervised learners for Computer Vision applications, encouraging robust visual feature representation learning.
- The Foundation Models achieve state-of-the-art performance on AD and TLE tasks across three operational viaducts, showing near-perfect accuracy on AD and significant improvement on evaluation metrics like R^2 score, MAE%, and MSE% for TLE tasks.

## Summary

, , , , 

Structural Health Monitoring (SHM) is crucial for ensuring the safety and reliability of civil infrastructures, particularly bridges and viaducts. In this paper, we introduce a novel approach using Transformer neural networks with a Masked Auto-Encoder architecture as Foundation Models for SHM. These models demonstrate the ability to learn generalizable representations from multiple large datasets through self-supervised pre-training, outperforming traditional methods on tasks such as Anomaly Detection (AD) and Traffic Load Estimation (TLE). We delve into the background of Transformer Neural Networks, highlighting their significance in NLP applications and deep learning. The Transformer architecture's innovation lies in self-attention mechanisms that capture long-range dependencies efficiently. We adapt an architecture derived from the Vision Transformer (ViT) for processing vibration data by breaking it into fixed-size patches and applying attention-based Transformers. Masked Autoencoders play a vital role in our approach, serving as scalable self-supervised learners for Computer Vision applications. By reconstructing image patches with missing information during training, these autoencoders encourage robust visual feature representation learning. Our specific masked autoencoder architecture is detailed to showcase its effectiveness in downstream tasks. Our Foundation Models achieve state-of-the-art performance on AD and TLE tasks across three operational viaducts. For AD, we achieve near-perfect accuracy within a short monitoring time span compared to traditional methods like Principal Component Analysis (PCA). On TLE tasks, our models outperform existing approaches significantly on evaluation metrics like R$^2$ score, MAE%, and MSE%. The paper is structured to provide necessary background information in Section 2 before delving into SHM literature overview in Section 3. We describe the viaducts considered for data collection and labeling processes in Section 4. Our foundation model approach is detailed in Section 5, covering the processing pipeline, model architecture, and training procedure. Experimental results are presented in Section 6 while conclusions are drawn in Section 7. Overall, our work showcases a promising direction for ML research in SHM by leveraging transformer-based masked autoencoders for effective anomaly detection and traffic load estimation. Future research avenues may include exploring larger pre-training datasets and optimizing model hyperparameters for enhanced performance. The code and pre-trained models are open-sourced for further exploration at https://github.com/eml-eda/tle-supervised.