# STARS: Zero-shot Sim-to-Real Transfer for Segmentation of Shipwrecks in Sonar Imagery

**Authors:** Advaith Venkatramanan Sethuraman, Katherine A. Skinner  
**arXiv:** [2310.01667](https://arxiv.org/abs/2310.01667)  
**Date:** 2023-10-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.01667v1/

## Key points

- Sethuraman and Skinner address the problem of sim-to-real transfer for object segmentation in shipwreck segmentation in side scan sonar imagery
- They propose a novel segmentation network called STARS for effective zero-shot sim-to-real transfer
- Their technical approach consists of synthetic data generation and network development and training
- Synthetic data is generated using physics-based rendering in Blender, with randomization of ship attributes
- The intensity of the sonar return is calculated using the SONAR equation
- The STARS network fuses a predicted deformation field and anomaly volume to improve generalization to real sonar images
- Evaluation on a real side scan sonar dataset shows a 20% increase in segmentation performance compared to baseline methods
- STARS achieves zero-shot sim-to-real transfer for shipwreck segmentation without fine-tuning on real data

## Summary

In this paper, Sethuraman and Skinner address the problem of sim-to-real transfer for object segmentation in the context of shipwreck segmentation in side scan sonar imagery. They propose a novel segmentation network called STARS, which aims to achieve effective zero-shot sim-to-real transfer for image segmentation when there are no real examples of the object of interest during training. The authors first describe their technical approach, which consists of two main components: synthetic data generation and network development and training. For synthetic data generation, they use physics-based rendering within the Blender graphics environment to generate synthetic side scan sonar data for training. They utilize 3D meshes of ships downloaded from TurboSquid and randomize acoustic reflectance, ship location, scale, and orientation. To calculate the intensity of the sonar return for each ray, they apply the SONAR equation considering sound propagation through water. Next, they explain how their STARS network addresses the challenge of sim-to-real transfer. The network fuses a predicted deformation field and anomaly volume to better generalize to real sonar images. This fusion allows STARS to achieve more effective zero-shot sim-to-real transfer for shipwreck segmentation. The authors evaluate the sim-to-real transfer capabilities of their method on a real side scan sonar dataset collected from field work surveys with an autonomous underwater vehicle (AUV). Importantly, STARS is trained entirely in simulation without any additional fine-tuning on real data. The results show that STARS provides a significant 20% increase in segmentation performance for shipwrecks compared to the best baseline method. This demonstrates its effectiveness in achieving zero-shot sim-to-real transfer for shipwreck segmentation in side scan sonar imagery. Overall, Sethuraman and Skinner's work presents an innovative approach to address the challenge of sim-to-real transfer in object segmentation. Their STARS network shows promising results in achieving accurate shipwreck segmentation without relying on real examples during training.