# Text Promptable Surgical Instrument Segmentation with Vision-Language Models

**Authors:** Zijian Zhou, Oluwatosin Alabi, Meng Wei, Tom Vercauteren, Miaojing Shi  
**arXiv:** [2306.09244](https://arxiv.org/abs/2306.09244)  
**Date:** 2023-06-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.09244v1/

## Key points

- Authors propose a novel approach for surgical instrument segmentation in minimally invasive surgeries
- Approach redefines the task as text promptable, enabling better comprehension and adaptability to new types of instruments
- Leveraging pretrained image and text encoders as model backbone
- Designing a text promptable mask decoder with attention- and convolution-based prompting schemes for surgical instrument segmentation prediction
- Introducing a hard instrument area reinforcement module to improve image feature comprehension and segmentation precision
- Model evaluated on EndoVis2017 and EndoVis2018 datasets, demonstrating superior performance and promising generalization capability
- Potential practical application in robotic-assisted surgery

## Summary

In this paper, the authors propose a novel approach for surgical instrument segmentation in minimally invasive surgeries. They address the challenges associated with the diversity and differentiation of surgical instruments by redefining the task as text promptable, enabling a more nuanced comprehension of instruments and adaptability to new types. Inspired by recent advancements in vision-language models, the authors leverage pretrained image and text encoders as their model backbone. They design a text promptable mask decoder consisting of attention- and convolution-based prompting schemes for surgical instrument segmentation prediction. This approach allows them to leverage multiple text prompts for each instrument through a new mixture of prompts mechanism, resulting in enhanced segmentation performance. To further improve image feature comprehension and segmentation precision, the authors introduce a hard instrument area reinforcement module. This module helps overcome the difficulties caused by imaging conditions within the surgical cavity. The proposed model is evaluated on EndoVis2017 and EndoVis2018 datasets, demonstrating superior performance and promising generalization capability. The results highlight the potential practical application of this promptable approach in robotic-assisted surgery. Overall, this paper presents the first implementation of a text promptable approach to surgical instrument segmentation. By leveraging vision-language models and incorporating multiple prompts, the authors achieve improved segmentation performance and offer significant potential for practical use in minimally invasive surgeries.