# TOLD: A Novel Two-Stage Overlap-Aware Framework for Speaker Diarization

**Authors:** Jiaming Wang, Zhihao Du, Shiliang Zhang  
**arXiv:** [2303.05397](https://arxiv.org/abs/2303.05397)  
**Date:** 2023-03-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.05397v1/

## Key points

- Introduction of end-to-end neural diarization (EEND) model
- Limitation of EEND as a multi-label prediction problem
- Proposal of overlap-aware EEND (EEND-OLA) approach
- Reformulation into single-label classification using power set encoding in EEND-OLA
- Development of Two-stage OverLap-aware Diarization framework (TOLD)
- Incorporation of speaker overlap-aware post-processing (SOAP) model in TOLD
- Results showing significant improvements with new approaches:
- EEND-OLA achieved a 14.39% relative improvement in diarization error rates (DER)
- SOAP provided an additional 19.33% relative improvement in performance
- TOLD method achieved a DER of 10.14% on the CALLHOME dataset, setting a new benchmark
- Potential for more accurate and efficient processing of audio data with multiple speakers speaking simultaneously due to advancements in overlap-aware frameworks for speaker diarization
- Contribution by Jiaming Wang, Zhihao Du, and Shiliang Zhang to the evolving field of speaker diarization technology

## Summary

In the field of speaker diarization, a recent advancement has been made with the introduction of the end-to-end neural diarization (EEND) model. This model has shown promising results particularly in scenarios where multiple speakers overlap in their speech. However, one limitation of EEND is that it treats speaker diarization as a multi-label prediction problem, where speaker activities are estimated independently without considering their dependencies. To address this issue, a new approach called overlap-aware EEND (EEND-OLA) has been proposed. In EEND-OLA, the traditional multi-label prediction problem is reformulated into a single-label classification problem using power set encoding. This enables the model to explicitly capture speaker overlaps and dependencies, leading to more accurate diarization results. Building upon the success of hybrid systems, researchers have further developed a Two-stage OverLap-aware Diarization framework (TOLD). This framework incorporates a speaker overlap-aware post-processing (SOAP) model to iteratively refine the diarization results obtained from EEND-OLA. Experimental evaluations have demonstrated significant improvements with these new approaches. Compared to the original EEND model, EEND-OLA achieved a remarkable 14.39% relative improvement in terms of diarization error rates (DER). Additionally, incorporating SOAP provided an additional 19.33% relative improvement in performance. As a result, the TOLD method achieved an impressive DER of 10.14% on the CALLHOME dataset, establishing a new state-of-the-art benchmark for speaker diarization tasks. Overall, these advancements in overlap-aware frameworks for speaker diarization showcase the potential for more accurate and efficient processing of audio data with multiple speakers speaking simultaneously. The research conducted by Jiaming Wang, Zhihao Du, and Shiliang Zhang represents a significant contribution to this evolving field and sets a new standard for future developments in speaker diarization technology.