AI summary · arXiv: 1910.14104v3 · eess.AS

End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation

Yi Luo, Zhuo Chen, Nima Mesgarani, Takuya Yoshioka
Original paper · DOI 10.1109/icassp40776.2020.9054177 Submitted to arXiv on 30 Oct. 2019 Published in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (May 2020) ICASSP 2020 License: CC BY-NC-SA 4.0 Also in: fr

How to read this summary? Illustrated guide

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
The summary is not ready yet

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.

AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts 220 papers already cite it: be emailed each new one. Citation alerts guide
Who works on it?

Among the 31 papers around this one, the name that comes back most: Xiao-Fei Li (4 papers)

10 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.

Continue with Google Create a free account: your first map is free
Weekly letter The latest letter of its field, "Computing beyond AI this week" of 16 November 2026: Computing beyond AI this week: The costs hidden in our maps, networks and algorithms · Receive one by email for free

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close Cites this paper
    Enhanced Deep Speech Separation in Clustered Ad Hoc Distributed Microphone Environments
    Jihyun Kim, Stijn Kindt, Nilesh Madhu et al. · 2024 · Interspeech · cited 5 times · Not summarized yet
    Abstract

    Ad-hoc distributed microphone environments, where microphone locations and numbers are unpredictable, present a challenge to traditional deep learning models, which typically require fixed architectures. To tailor deep learning models to accommodate arbitrary array configurations, the Transform-Average-Concatenate (TAC) layer was previously introduced. In this work, we integrate TAC layers with dual-path transformers for speech separation from two simultaneous talkers in realistic settings. However, the distributed nature makes it hard to fuse information across microphones efficiently. Therefore, we explore the efficacy of blindly clustering microphones around sources of interest prior to e…

  2. Very close Cites this paper
    Continuous Speech Separation with Ad Hoc Microphone Arrays
    Dong-Mei Wang, T. Yoshioka, Zhuo Chen et al. · 2021 · European Signal Processing Conference · cited 33 times · Not summarized yet
    Abstract

    Speech separation has been shown effective for multi-talker speech recognition. Under the ad hoc microphone array setup where the array consists of spatially distributed asynchronous microphones, additional challenges must be overcome as the geometry and number of microphones are unknown beforehand. Prior studies show, with a spatial-temporal-interleaving structure, neural networks can efficiently utilize the multi-channel signals of the ad hoc array. In this paper, we further extend this approach to continuous speech separation. Several techniques are introduced to enable speech separation for real continuous recordings. First, we apply a transformer-based network for spatio-temporal modeli…

  3. Very close Cites this paper Uses this paper in depth
    Multi-Channel Narrow-Band Deep Speech Separation with Full-Band Permutation Invariant Training
    Chang-Sheng Quan, Xiao-Fei Li · 2021 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cited 23 times · Not summarized yet
    Abstract

    This paper addresses the problem of multi-channel multi-speech separation based on deep learning techniques. In the short time Fourier transform domain, we propose an end-to-end narrow-band network that directly takes as input the multi-channel mixture signals of one frequency, and outputs the separated signals of this frequency. In narrow-band, the spatial information (or inter-channel difference) can well discriminate between speakers at different positions. This information is intensively used in many narrow-band speech separation methods, such as beamforming and clustering of spatial vectors. The proposed network is trained to learn a rule to automatically exploit this information and pe…

  4. Very close Cites this paper Uses this paper in depth
    Implicit Filter-and-sum Network for Multi-channel Speech Separation
    Yi Luo, N. Mesgarani · 2020 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
    Abstract

    Various neural network architectures have been proposed in recent years for the task of multi-channel speech separation. Among them, the filter-and-sum network (FaSNet) performs end-to-end time-domain filter-and-sum beamforming and has shown effective in both ad-hoc and fixed microphone array geometries. In this paper, we investigate multiple ways to improve the performance of FaSNet. From the problem formulation perspective, we change the explicit time-domain filter-and-sum operation which involves all the microphones into an implicit filter-and-sum operation in the latent space of only the reference microphone. The filter-and-sum operation is applied on a context around the frame to be sep…

  5. Very close Cites this paper
    Efficient, Cluster-Informed, Deep Speech Separation with Cross-Cluster Information in AD-HOC Wireless Acoustic Sensor Networks
    Stijn Kindt, Jihyun Kim, Hong-Goo Kang et al. · 2024 · International Workshop on Acoustic Signal Enhancement · cited 1 time · Not summarized yet
    Abstract

    Environments with ad hoc distributed microphones, characterised by unpredictable locations and varying numbers, pose a significant challenge to most conventional deep-learning based speech enhancement and separation models. Transform-Average-Concatenate (TAC) layers in combination with dual-path transformer (DPT) models have been proposed for speech separation with flexible array configurations. For widely distributed microphone setups, we previously showed that blindly clustering microphones around target sound sources and processing each cluster separately yields good separation. In this work, we propose to further improve output signal quality by exploiting inter-cluster information by su…

  6. Very close Cites this paper
    Multi-Channel Talker-Independent Speaker Separation Through Location-Based Training
    H. Taherian, Ke Tan, De-Liang Wang · 2022 · IEEE/ACM Transactions on Audio Speech and Language Processing · cited 42 times · Not summarized yet
    Abstract

    Permutation ambiguity is a crucial issue for deep learning based talker-independent speaker separation. Deep clustering and permutation invariant training (PIT) have been widely used to address the permutation ambiguity problem in monaural scenarios. Although both approaches have been extended to multi-microphone scenarios, we believe that the permutation ambiguity problem can be naturally avoided by leveraging the spatial relations of multiple speakers. In this article, we present location-based training (LBT), a new approach to achieve talker independency in multi-channel speaker separation. Unlike PIT that examines all possible permutations, LBT assigns speakers according to their positio…

  7. Close Cites this paper
    Implicit Filter-and-Sum Network for End-to-End Multi-Channel Speech Separation
    Yi Luo, N. Mesgarani · 2021 · Interspeech · cited 18 times · Not summarized yet
    Abstract

    Various neural network architectures have been proposed in recent years for the task of multi-channel speech separation. Among them, the filter-and-sum network (FaSNet) performs end-to-end time-domain filter-and-sum beamforming and has shown effective in both ad-hoc and fixed microphone array geometries. However, whether such explicit beamforming operation is a necessary and valid formulation remains unclear. In this paper, we investigate the beamforming operation and show that it is not necessary. To further improve the performance, we change the explicit waveform-level filter-and-sum operation into an implicit feature-level filter-and-sum operation around a context of features. A feature-level …

  8. Close Cites this paper
    Multi-Channel Speech Separation with Cross-Attention and Beamforming
    Ladislav Mošner, O. Plchot, Jun-Yi Peng et al. · 2023 · Interspeech · cited 1 time · Not summarized yet
    Abstract

    Originally, single-channel source separation gained more research interest. It resulted in immense progress. Multi-channel (MC) separation comes with new challenges posed by adverse indoor conditions making it an important field of study. We seek to combine promising ideas from the two worlds. First, we build MC models by extending current single-channel time-domain separators relying on their strength. Our approach allows reusing pre-trained models by inserting designed lightweight reference channel attention (RCA) combiner, the only trained module. It comprises two blocks: the former allows attending to different parts of other channels w.r.t. the reference one, and the latter provides an …

Prior work it builds on

  1. Close Used in depth by this paper A method it uses
    Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
    Yi Luo, Nima Mesgarani · 2018 · Sound · cited 2,311 times · AI summary available
    Abstract

    Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time-frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time-frequency representation for speech separation, and the long latency in calculating the spectrograms. To address these shortcomings, we propose a fully-convolutional time-domain audio separation network (Conv-TasNet), a deep learning framework for en…

  2. Close Used in depth by this paper A method it uses
    FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio Processing
    Yi Luo, Cong Han, N. Mesgarani et al. · 2019 · Automatic Speech Recognition & Understanding · cited 195 times · Not summarized yet
    Abstract

    1. ABSTRACT Beamforming has been extensively investigated for multi-channel audio processing tasks. Recently, learning-based beamforming methods, sometimes called neural beamformers, have achieved significant improvements in both signal quality (e.g. signal-to-noise ratio (SNR)) and speech recognition (e.g. word error rate (WER)). Such systems are generally non-causal and require a large context for robust estimation of inter-channel features, which is impractical in applications requiring low-latency responses. In this paper, we propose filter-and-sum network (FaSNet), a time-domain, filter-based beamforming approach suitable for low-latency scenarios. FaSNet has a two-stage system design t…

  3. Close
    Unified Architecture for Multichannel End-to-End Speech Recognition With Neural Beamforming
    Tsubasa Ochiai, Shinji Watanabe, T. Hori et al. · 2017 · IEEE Journal on Selected Topics in Signal Processing · cited 95 times · Not summarized yet
    Abstract

    This paper proposes a unified architecture for end-to-end automatic speech recognition (ASR) to encompass microphone-array signal processing such as a state-of-the-art neural beamformer within the end-to-end framework. Recently, the end-to-end ASR paradigm has attracted great research interest as an alternative to conventional hybrid paradigms with deep neural networks and hidden Markov models. Using this novel paradigm, we simplify ASR architecture by integrating such ASR components as acoustic, phonetic, and language models with a single neural network and optimize the overall components for the end-to-end ASR objective: generating a correct label sequence. Although most existing end-to-en…

  4. Close
    Multichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition
    T. Sainath, Ron J. Weiss, K. Wilson et al. · 2017 · IEEE/ACM Transactions on Audio Speech and Language Processing · cited 214 times · Not summarized yet
    Abstract

    Multichannel automatic speech recognition (ASR) systems commonly separate speech enhancement, including localization, beamforming, and postfiltering, from acoustic modeling. In this paper, we perform multichannel enhancement jointly with acoustic modeling in a deep neural network framework. Inspired by beamforming, which leverages differences in the fine time structure of the signal at different microphones to filter energy arriving from different directions, we explore modeling the raw time-domain waveform directly. We introduce a neural network architecture, which performs multichannel filtering in the first layer of the network, and show that this network learns to be robust to varying ta…

  5. Close
    Deep long short-term memory adaptive beamforming networks for multichannel robust speech recognition
    Zhong Meng, Shinji Watanabe, J. Hershey et al. · 2017 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cited 89 times · Not summarized yet
    Abstract

    Far-field speech recognition in noisy and reverberant conditions remains a challenging problem despite recent deep learning breakthroughs. This problem is commonly addressed by acquiring a speech signal from multiple microphones and performing beamforming over them. In this paper, we propose to use a recurrent neural network with long short-term memory (LSTM) architecture to adaptively estimate real-time beamforming filter coefficients to cope with non-stationary environmental noise and dynamic nature of source and microphones positions which results in a set of timevarying room impulse responses. The LSTM adaptive beamformer is jointly trained with a deep LSTM acoustic model to predict seno…

Papers that cite it

  1. Close Uses this paper in depth
    Multichannel Speech Separation with Narrow-band Conformer
    Chang-Sheng Quan, Xiao-Fei Li · 2022 · Interspeech · cited 19 times · Not summarized yet
    Abstract

    This work proposes a multichannel speech separation method with narrow-band Conformer (named NBC). The network is trained to learn to automatically exploit narrow-band speech separation information, such as spatial vector clustering of multiple speakers. Specifically, in the short-time Fourier transform (STFT) domain, the network processes each frequency independently, and is shared by all frequencies. For one frequency, the network inputs the STFT coefficients of multichannel mixture signals, and predicts the STFT coefficients of separated speech signals. Clustering of spatial vectors shares a similar principle with the self-attention mechanism in the sense of computing the similarity of ve…

  2. Close Uses this paper in depth
    Learnable Cross-Correlation based Filter-and-Sum Networks for Multi-channel Speech Separation
    Xian-Rui Wang, Shiqi Zhang, Bo He et al. · 2024 · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference · cited 1 time · Not summarized yet
    Abstract

    Multichannel source separation plays an important role in audio and speech signal processing. With recent advancements in deep neural networks (DNN), numerous DNN-based beamforming algorithms have been developed. To leverage spatial information, a time domain filter-and-sum network (FaSNet) was introduced, and the transform average concatenate (TAC) technique was subsequently adopted to further enhance separation performance. FaSNet captures spatial information by assessing cosine similarity between different channels; but this approach may have limited spatial resolution and could exhibit bias in noisy, reverberant environments, thereby potentially compromising performance. Motivated by the…

  3. Close Uses this paper in depth
    Microphone Array Generalization for Multichannel Narrowband Deep Speech Enhancement
    Siyuan Zhang, Xiao-Fei Li · 2021 · Interspeech · cited 21 times · Not summarized yet
    Abstract

    This paper addresses the problem of microphone array generalization for deep-learning-based end-to-end multichannel speech enhancement. We aim to train a unique deep neural network (DNN) potentially performing well on unseen microphone arrays. The microphone array geometry shapes the network's parameters when training on a fixed microphone array, and thus restricts the generalization of the trained network to another microphone array. To resolve this problem, a single network is trained using data recorded by various microphone arrays of different geometries. We design three variants of our recently proposed narrowband network to cope with the agnostic number of microphones. Overall, the goa…

  4. Close Uses this paper in depth
    SDNet: Stream-attention and Dual-feature Learning Network for Ad-hoc Array Speech Separation
    Honglong Wang, Chengyun Deng, Yan-Jie Fu et al. · 2023 · Interspeech · cited 3 times · Not summarized yet
    Abstract

    Considerable progress has been made in multi-channel speech separation for fixed arrays. In this paper, we aim to develop a robust system for ad-hoc arrays to deal with uncertainties of microphone locations and numbers. Previous works commonly used the averaging method for ad-hoc arrays, overlooking the diversity of microphones in various positions. Some studies suggest that microphones with high signal-to-noise ratio(SNR) are more helpful in improving speech quality. Motivated by this, we propose stream-attention and dual-feature learning network called SDNet. The key points are as follows: 1) We propose a dual-feature learning block with fewer parameters to learn the long-term dependency b…

  5. Close* Uses this paper in depth
    A New Neural Beamformer for Multi-channel Speech Separation
    Ruqiao Liu, Yi Zhou, Hong-Qing Liu et al. · 2022 · Journal of Signal Processing Systems · cited 5 times · Not summarized yet
How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 5 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.