AI summary · arXiv: 1910.14104v3 · eess.AS
End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 31 papers around this one, the name that comes back most: Xiao-Fei Li (4 papers)
10 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Enhanced Deep Speech Separation in Clustered Ad Hoc Distributed Microphone EnvironmentsJihyun Kim, Stijn Kindt, Nilesh Madhu et al. · 2024 · Interspeech · cited 5 times · Not summarized yet
Abstract
Ad-hoc distributed microphone environments, where microphone locations and numbers are unpredictable, present a challenge to traditional deep learning models, which typically require fixed architectures. To tailor deep learning models to accommodate arbitrary array configurations, the Transform-Average-Concatenate (TAC) layer was previously introduced. In this work, we integrate TAC layers with dual-path transformers for speech separation from two simultaneous talkers in realistic settings. However, the distributed nature makes it hard to fuse information across microphones efficiently. Therefore, we explore the efficacy of blindly clustering microphones around sources of interest prior to e…
-
Continuous Speech Separation with Ad Hoc Microphone ArraysDong-Mei Wang, T. Yoshioka, Zhuo Chen et al. · 2021 · European Signal Processing Conference · cited 33 times · Not summarized yet
Abstract
Speech separation has been shown effective for multi-talker speech recognition. Under the ad hoc microphone array setup where the array consists of spatially distributed asynchronous microphones, additional challenges must be overcome as the geometry and number of microphones are unknown beforehand. Prior studies show, with a spatial-temporal-interleaving structure, neural networks can efficiently utilize the multi-channel signals of the ad hoc array. In this paper, we further extend this approach to continuous speech separation. Several techniques are introduced to enable speech separation for real continuous recordings. First, we apply a transformer-based network for spatio-temporal modeli…
-
Multi-Channel Narrow-Band Deep Speech Separation with Full-Band Permutation Invariant TrainingChang-Sheng Quan, Xiao-Fei Li · 2021 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cited 23 times · Not summarized yet
Abstract
This paper addresses the problem of multi-channel multi-speech separation based on deep learning techniques. In the short time Fourier transform domain, we propose an end-to-end narrow-band network that directly takes as input the multi-channel mixture signals of one frequency, and outputs the separated signals of this frequency. In narrow-band, the spatial information (or inter-channel difference) can well discriminate between speakers at different positions. This information is intensively used in many narrow-band speech separation methods, such as beamforming and clustering of spatial vectors. The proposed network is trained to learn a rule to automatically exploit this information and pe…
-
Implicit Filter-and-sum Network for Multi-channel Speech SeparationYi Luo, N. Mesgarani · 2020 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
Abstract
Various neural network architectures have been proposed in recent years for the task of multi-channel speech separation. Among them, the filter-and-sum network (FaSNet) performs end-to-end time-domain filter-and-sum beamforming and has shown effective in both ad-hoc and fixed microphone array geometries. In this paper, we investigate multiple ways to improve the performance of FaSNet. From the problem formulation perspective, we change the explicit time-domain filter-and-sum operation which involves all the microphones into an implicit filter-and-sum operation in the latent space of only the reference microphone. The filter-and-sum operation is applied on a context around the frame to be sep…
-
Efficient, Cluster-Informed, Deep Speech Separation with Cross-Cluster Information in AD-HOC Wireless Acoustic Sensor NetworksStijn Kindt, Jihyun Kim, Hong-Goo Kang et al. · 2024 · International Workshop on Acoustic Signal Enhancement · cited 1 time · Not summarized yet
Abstract
Environments with ad hoc distributed microphones, characterised by unpredictable locations and varying numbers, pose a significant challenge to most conventional deep-learning based speech enhancement and separation models. Transform-Average-Concatenate (TAC) layers in combination with dual-path transformer (DPT) models have been proposed for speech separation with flexible array configurations. For widely distributed microphone setups, we previously showed that blindly clustering microphones around target sound sources and processing each cluster separately yields good separation. In this work, we propose to further improve output signal quality by exploiting inter-cluster information by su…
-
Multi-Channel Talker-Independent Speaker Separation Through Location-Based TrainingH. Taherian, Ke Tan, De-Liang Wang · 2022 · IEEE/ACM Transactions on Audio Speech and Language Processing · cited 42 times · Not summarized yet
Abstract
Permutation ambiguity is a crucial issue for deep learning based talker-independent speaker separation. Deep clustering and permutation invariant training (PIT) have been widely used to address the permutation ambiguity problem in monaural scenarios. Although both approaches have been extended to multi-microphone scenarios, we believe that the permutation ambiguity problem can be naturally avoided by leveraging the spatial relations of multiple speakers. In this article, we present location-based training (LBT), a new approach to achieve talker independency in multi-channel speaker separation. Unlike PIT that examines all possible permutations, LBT assigns speakers according to their positio…
-
Implicit Filter-and-Sum Network for End-to-End Multi-Channel Speech SeparationYi Luo, N. Mesgarani · 2021 · Interspeech · cited 18 times · Not summarized yet
Abstract
Various neural network architectures have been proposed in recent years for the task of multi-channel speech separation. Among them, the filter-and-sum network (FaSNet) performs end-to-end time-domain filter-and-sum beamforming and has shown effective in both ad-hoc and fixed microphone array geometries. However, whether such explicit beamforming operation is a necessary and valid formulation remains unclear. In this paper, we investigate the beamforming operation and show that it is not necessary. To further improve the performance, we change the explicit waveform-level filter-and-sum operation into an implicit feature-level filter-and-sum operation around a context of features. A feature-level …
-
Multi-Channel Speech Separation with Cross-Attention and BeamformingLadislav Mošner, O. Plchot, Jun-Yi Peng et al. · 2023 · Interspeech · cited 1 time · Not summarized yet
Abstract
Originally, single-channel source separation gained more research interest. It resulted in immense progress. Multi-channel (MC) separation comes with new challenges posed by adverse indoor conditions making it an important field of study. We seek to combine promising ideas from the two worlds. First, we build MC models by extending current single-channel time-domain separators relying on their strength. Our approach allows reusing pre-trained models by inserting designed lightweight reference channel attention (RCA) combiner, the only trained module. It comprises two blocks: the former allows attending to different parts of other channels w.r.t. the reference one, and the latter provides an …
Prior work it builds on
-
Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech SeparationYi Luo, Nima Mesgarani · 2018 · Sound · cited 2,311 times · AI summary available
Abstract
Single-channel, speaker-independent speech separation methods have recently seen great progress. However, the accuracy, latency, and computational cost of such methods remain insufficient. The majority of the previous methods have formulated the separation problem through the time-frequency representation of the mixed signal, which has several drawbacks, including the decoupling of the phase and magnitude of the signal, the suboptimality of time-frequency representation for speech separation, and the long latency in calculating the spectrograms. To address these shortcomings, we propose a fully-convolutional time-domain audio separation network (Conv-TasNet), a deep learning framework for en…
-
FaSNet: Low-Latency Adaptive Beamforming for Multi-Microphone Audio ProcessingYi Luo, Cong Han, N. Mesgarani et al. · 2019 · Automatic Speech Recognition & Understanding · cited 195 times · Not summarized yet
Abstract
1. ABSTRACT Beamforming has been extensively investigated for multi-channel audio processing tasks. Recently, learning-based beamforming methods, sometimes called neural beamformers, have achieved significant improvements in both signal quality (e.g. signal-to-noise ratio (SNR)) and speech recognition (e.g. word error rate (WER)). Such systems are generally non-causal and require a large context for robust estimation of inter-channel features, which is impractical in applications requiring low-latency responses. In this paper, we propose filter-and-sum network (FaSNet), a time-domain, filter-based beamforming approach suitable for low-latency scenarios. FaSNet has a two-stage system design t…
-
Unified Architecture for Multichannel End-to-End Speech Recognition With Neural BeamformingTsubasa Ochiai, Shinji Watanabe, T. Hori et al. · 2017 · IEEE Journal on Selected Topics in Signal Processing · cited 95 times · Not summarized yet
Abstract
This paper proposes a unified architecture for end-to-end automatic speech recognition (ASR) to encompass microphone-array signal processing such as a state-of-the-art neural beamformer within the end-to-end framework. Recently, the end-to-end ASR paradigm has attracted great research interest as an alternative to conventional hybrid paradigms with deep neural networks and hidden Markov models. Using this novel paradigm, we simplify ASR architecture by integrating such ASR components as acoustic, phonetic, and language models with a single neural network and optimize the overall components for the end-to-end ASR objective: generating a correct label sequence. Although most existing end-to-en…
-
Multichannel Signal Processing With Deep Neural Networks for Automatic Speech RecognitionT. Sainath, Ron J. Weiss, K. Wilson et al. · 2017 · IEEE/ACM Transactions on Audio Speech and Language Processing · cited 214 times · Not summarized yet
Abstract
Multichannel automatic speech recognition (ASR) systems commonly separate speech enhancement, including localization, beamforming, and postfiltering, from acoustic modeling. In this paper, we perform multichannel enhancement jointly with acoustic modeling in a deep neural network framework. Inspired by beamforming, which leverages differences in the fine time structure of the signal at different microphones to filter energy arriving from different directions, we explore modeling the raw time-domain waveform directly. We introduce a neural network architecture, which performs multichannel filtering in the first layer of the network, and show that this network learns to be robust to varying ta…
-
Deep long short-term memory adaptive beamforming networks for multichannel robust speech recognitionZhong Meng, Shinji Watanabe, J. Hershey et al. · 2017 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cited 89 times · Not summarized yet
Abstract
Far-field speech recognition in noisy and reverberant conditions remains a challenging problem despite recent deep learning breakthroughs. This problem is commonly addressed by acquiring a speech signal from multiple microphones and performing beamforming over them. In this paper, we propose to use a recurrent neural network with long short-term memory (LSTM) architecture to adaptively estimate real-time beamforming filter coefficients to cope with non-stationary environmental noise and dynamic nature of source and microphones positions which results in a set of timevarying room impulse responses. The LSTM adaptive beamformer is jointly trained with a deep LSTM acoustic model to predict seno…
Papers that cite it
-
Multichannel Speech Separation with Narrow-band ConformerChang-Sheng Quan, Xiao-Fei Li · 2022 · Interspeech · cited 19 times · Not summarized yet
Abstract
This work proposes a multichannel speech separation method with narrow-band Conformer (named NBC). The network is trained to learn to automatically exploit narrow-band speech separation information, such as spatial vector clustering of multiple speakers. Specifically, in the short-time Fourier transform (STFT) domain, the network processes each frequency independently, and is shared by all frequencies. For one frequency, the network inputs the STFT coefficients of multichannel mixture signals, and predicts the STFT coefficients of separated speech signals. Clustering of spatial vectors shares a similar principle with the self-attention mechanism in the sense of computing the similarity of ve…
-
Learnable Cross-Correlation based Filter-and-Sum Networks for Multi-channel Speech SeparationXian-Rui Wang, Shiqi Zhang, Bo He et al. · 2024 · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference · cited 1 time · Not summarized yet
Abstract
Multichannel source separation plays an important role in audio and speech signal processing. With recent advancements in deep neural networks (DNN), numerous DNN-based beamforming algorithms have been developed. To leverage spatial information, a time domain filter-and-sum network (FaSNet) was introduced, and the transform average concatenate (TAC) technique was subsequently adopted to further enhance separation performance. FaSNet captures spatial information by assessing cosine similarity between different channels; but this approach may have limited spatial resolution and could exhibit bias in noisy, reverberant environments, thereby potentially compromising performance. Motivated by the…
-
Microphone Array Generalization for Multichannel Narrowband Deep Speech EnhancementSiyuan Zhang, Xiao-Fei Li · 2021 · Interspeech · cited 21 times · Not summarized yet
Abstract
This paper addresses the problem of microphone array generalization for deep-learning-based end-to-end multichannel speech enhancement. We aim to train a unique deep neural network (DNN) potentially performing well on unseen microphone arrays. The microphone array geometry shapes the network's parameters when training on a fixed microphone array, and thus restricts the generalization of the trained network to another microphone array. To resolve this problem, a single network is trained using data recorded by various microphone arrays of different geometries. We design three variants of our recently proposed narrowband network to cope with the agnostic number of microphones. Overall, the goa…
-
SDNet: Stream-attention and Dual-feature Learning Network for Ad-hoc Array Speech SeparationHonglong Wang, Chengyun Deng, Yan-Jie Fu et al. · 2023 · Interspeech · cited 3 times · Not summarized yet
Abstract
Considerable progress has been made in multi-channel speech separation for fixed arrays. In this paper, we aim to develop a robust system for ad-hoc arrays to deal with uncertainties of microphone locations and numbers. Previous works commonly used the averaging method for ad-hoc arrays, overlooking the diversity of microphones in various positions. Some studies suggest that microphones with high signal-to-noise ratio(SNR) are more helpful in improving speech quality. Motivated by this, we propose stream-attention and dual-feature learning network called SDNet. The key points are as follows: 1) We propose a dual-feature learning block with fewer parameters to learn the long-term dependency b…
-
A New Neural Beamformer for Multi-channel Speech SeparationRuqiao Liu, Yi Zhou, Hong-Qing Liu et al. · 2022 · Journal of Signal Processing Systems · cited 5 times · Not summarized yet
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 5 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.