Filler Word Detection and Classification: A Dataset and Benchmark

Ge Zhu; Juan-Pablo Caceres; Justin Salamon

AI summary · arXiv: 2203.15135v1 · cs.CL

Filler Word Detection and Classification: A Dataset and Benchmark

Ge Zhu, Juan-Pablo Caceres, Justin Salamon

Original paper Submitted to arXiv on 28 Mar. 2022 Submitted to Insterspeech 2022 License: CC BY-NC-ND 4.0

How to read this summary? Illustrated guide

The essentials

Authors Ge Zhu, Juan-Pablo Caceres, and Justin Salamon address identifying and categorizing filler words in speech recordings
Introduce PodcastFillers dataset with 35K annotated filler words and 50K annotations of other common sounds in podcasts
Propose a pipeline combining Voice Activity Detection (VAD) and Automatic Speech Recognition (ASR) for identifying filler candidates
Evaluate pipeline on PodcastFillers, showing ASR significantly improves detection accuracy
Achieve state-of-the-art results in detecting and classifying filler words
Make PodcastFillers publicly available to establish a benchmark for future research

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Filler Word Detection

Read further

Comprehensive Summary
Layman's Summary
Blog article
Original abstract

, , , , In their work titled "Filler Word Detection and Classification: A Dataset and Benchmark," authors Ge Zhu, Juan-Pablo Caceres, and Justin Salamon address the issue of identifying and categorizing filler words like 'uh' or 'um' in speech recordings. These filler words are commonly used as pauses during speech, making their removal a tedious task in media editing. The authors highlight the lack of research in this area due to the absence of a comprehensive dataset with annotated filler words for training and evaluation. To bridge this gap, the authors introduce a new speech dataset called PodcastFillers, containing 35K annotated filler words along with 50K annotations of other common sounds found in podcasts such as breaths, laughter, and word repetitions. They propose a pipeline that combines Voice Activity Detection (VAD) and Automatic Speech Recognition (ASR) to identify potential filler candidates and a classifier to differentiate between different types of filler words. The authors evaluate their pipeline on PodcastFillers, comparing it against several baseline methods and conducting a detailed ablation study to assess the impact of using ASR compared to transcription-free approaches like keyword spotting. Their results demonstrate that leveraging ASR significantly improves detection accuracy, outperforming keyword spotting techniques. The proposed pipeline achieves state-of-the-art results in detecting and classifying filler words. By making PodcastFillers publicly available, the authors aim to establish a benchmark for future research in this field. Their work not only contributes to enhancing media editing processes by automating filler word detection but also sheds light on the importance of utilizing ASR technology for improved performance in speech analysis tasks. This study serves as a valuable resource for researchers interested in developing more efficient methods for identifying and managing filler words in audio recordings.

Introduction

Filler words are a common occurrence in everyday speech, often used as pauses or placeholders while we gather our thoughts. These words, such as 'uh' or 'um', may seem insignificant but can add up to significant amounts of time in media recordings. As a result, removing filler words has become an essential task in media editing processes. However, this task is tedious and time-consuming, highlighting the need for automated methods to detect and remove filler words.

In their research paper titled "Filler Word Detection and Classification: A Dataset and Benchmark," authors Ge Zhu, Juan-Pablo Caceres, and Justin Salamon address this issue by introducing a new dataset called PodcastFillers along with a pipeline for detecting and classifying filler words. This article will provide a detailed overview of their work.

The Need for Research on Filler Words

Despite the prevalence of filler words in speech recordings, there has been limited research on identifying and categorizing them. One reason for this is the lack of comprehensive datasets with annotated filler words that can be used for training and evaluation purposes. Existing datasets either have limited annotations or focus on specific types of fillers like laughter or breaths.

The absence of a benchmark dataset makes it challenging to compare different approaches for detecting filler words accurately. Moreover, most studies rely on keyword spotting techniques rather than leveraging more advanced technologies like Automatic Speech Recognition (ASR). Therefore, there is a need for research that utilizes ASR technology to improve detection accuracy.

The PodcastFillers Dataset

To address these gaps in research, Zhu et al. introduce PodcastFillers – a new dataset containing 35K annotated filler word instances from over 500 hours of podcast audio recordings. The dataset also includes annotations for other common sounds found in podcasts such as breaths, laughter, word repetitions etc., making it suitable not only for filler word detection but also for other speech analysis tasks.

The authors collected the dataset by manually annotating filler words in podcast recordings from various sources, including popular podcasts and online repositories. They ensured a diverse range of speakers, accents, and topics to make the dataset representative of real-world scenarios. The annotations were verified by multiple annotators to ensure accuracy.

The Proposed Pipeline

Zhu et al.'s pipeline consists of two main components – Voice Activity Detection (VAD) and Automatic Speech Recognition (ASR). VAD is used to identify potential filler candidates by detecting silence segments in audio recordings. These segments are then passed through an ASR system, which transcribes them into text. The final step involves using a classifier to differentiate between different types of fillers based on their transcription.

To train the classifier, the authors use PodcastFillers along with existing datasets like LibriSpeech and CommonVoice. They compare several baseline methods such as keyword spotting and VAD-only approaches against their proposed pipeline to evaluate its performance.

Results and Evaluation

The results demonstrate that leveraging ASR significantly improves detection accuracy compared to keyword spotting techniques. The proposed pipeline achieves state-of-the-art results in detecting and classifying filler words, outperforming all baseline methods. Additionally, the authors conduct an ablation study to assess the impact of using ASR compared to transcription-free approaches like keyword spotting.

Conclusion

In conclusion, Zhu et al.'s work makes significant contributions towards automating filler word detection in speech recordings. By introducing PodcastFillers – a comprehensive dataset with annotated filler words – they provide a benchmark for future research in this field. Their proposed pipeline leverages advanced technologies like ASR to achieve state-of-the-art results in identifying and categorizing fillers accurately. This study not only enhances media editing processes but also highlights the importance of utilizing ASR technology for improved performance in speech analysis tasks.

Created on 08 Dec. 2024

Was this synthesis useful?

Score: 0

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

⚠The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.

AI assistant instructions?

Tools for this paper

Explain it differently 3 free rewrites left today

Save to a collection Group papers in collections to compare them, chat with them and export them. Create a free account Log in

Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Markdown (Notion, Obsidian)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero

Citation alerts Citation alerts guide

Who works on it? The people and institutions who publish on its topic

Journal club Premium+ Slides to present it

Versions What changed between versions

Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

Related
Artificial Disfluency Detection, Uh No, Disfluency Generation for the Masses
T. Passali, T. Mavropoulos, G. Tsoumakas et al. · 2022 · Computation and Language · AI summary available

Abstract
Existing approaches for disfluency detection typically require the existence of large annotated datasets. However, current datasets for this task are limited, suffer from class imbalance, and lack some types of disfluencies that can be encountered in real-world scenarios. This work proposes LARD, a method for automatically generating artificial disfluencies from fluent text. LARD can simulate all the different types of disfluencies (repetitions, replacements and restarts) based on the reparandum/interregnum annotation scheme. In addition, it incorporates contextual embeddings into the disfluency generation to produce realistic context-aware artificial disfluencies. Since the proposed method …
Related
Efficient Training of Language Models to Fill in the Middle
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak et al. · 2022 · Computation and Language · AI summary available

Abstract
We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data augmentation has garnered much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not harm the original left-to-right generative capability, as measured by perplexity and sampling evaluations across a wide range of scales. Given the usefulness, simplicity, and efficiency of training models to fill-in-the-middle (FIM), we suggest that future autoregressive language models be trained with …
Related
Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics
Siddhant Arora, Zhiyun Lu, Chung-Cheng Chiu et al. · 2025 · Computation and Language · AI summary available

Abstract
The recent wave of audio foundation models (FMs) could provide new capabilities for conversational modeling. However, there have been limited efforts to evaluate these audio FMs comprehensively on their ability to have natural and interactive conversations. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns without too much overlapping speech or long stretches of silence. Inspired by this, we ask whether the recently proposed audio FMs can understand, predict, and perform turn-taking events? To answer this, we propose a novel evaluation protocol that can assess spoken dialog system's turn-taking capabilities usin…
Related
Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods
Potsawee Manakul, Mark J. F. Gales · 2022 · Computation and Language · AI summary available

Abstract
Automatic summary assessment is useful for both machine-generated and human-produced summaries. Automatically evaluating the summary text given the document enables, for example, summary generation system development and detection of inappropriate summaries. Summary assessment can be run in a number of modes: ranking summary generation systems; ranking summaries of a particular document; and estimating the quality of a document-summary pair on an absolute scale. Existing datasets with annotation for summary assessment are usually based on news summarization datasets such as CNN/DailyMail or XSum. In this work, we describe a new dataset, the podcast summary assessment corpus, a collection of …
Related
Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?
Dominic Petrak, Nafise Sadat Moosavi, Ye Tian et al. · 2023 · Computation and Language · AI summary available

Abstract
Learning from free-text human feedback is essential for dialog systems, but annotated data is scarce and usually covers only a small fraction of error types known in conversational AI. Instead of collecting and annotating new datasets from scratch, recent advances in synthetic dialog generation could be used to augment existing dialog datasets with the necessary annotations. However, to assess the feasibility of such an effort, it is important to know the types and frequency of free-text human feedback included in these datasets. In this work, we investigate this question for a variety of commonly used dialog datasets, including MultiWoZ, SGD, BABI, PersonaChat, Wizards-of-Wikipedia, and the…
Related
XferBench: a Data-Driven Benchmark for Emergent Language
Brendon Boldt, David Mortensen · 2024 · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human · AI summary available

Abstract
In this paper, we introduce a benchmark for evaluating the overall quality of emergent languages using data-driven methods. Specifically, we interpret the notion of the "quality" of an emergent language as its similarity to human language within a deep learning framework. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language -- the better the downstream performance, the better the emergent language. We implement this benchmark as an easy-to-use Python package that only requires a text file of utterances from the emergent language to be evaluated. Finally, we empirically test the benchmark's validity using human, synthetic, and emergen…
Related
Proficiency assessment of L2 spoken English using wav2vec 2.0
Stefano Bannò, Marco Matassoni · 2022 · Computation and Language · AI summary available

Abstract
The increasing demand for learning English as a second language has led to a growing interest in methods for automatically assessing spoken language proficiency. Most approaches use hand-crafted features, but their efficacy relies on their particular underlying assumptions and they risk discarding potentially salient information about proficiency. Other approaches rely on transcriptions produced by ASR systems which may not provide a faithful rendition of a learner's utterance in specific scenarios (e.g., non-native children's spontaneous speech). Furthermore, transcriptions do not yield any information about relevant aspects such as intonation, rhythm or prosody. In this paper, we investiga…
Related
SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
Xin Zhang, Dong Zhang, Shimin Li et al. · 2023 · Computation and Language · AI summary available

Abstract
Current speech large language models build upon discrete speech representations, which can be categorized into semantic tokens and acoustic tokens. However, existing speech tokens are not specifically designed for speech language modeling. To assess the suitability of speech tokens for building speech language models, we established the first benchmark, SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are ideal for this purpose. Therefore, we propose SpeechTokenizer, a unified speech tokenizer for speech large language models. SpeechTokenizer adopts the Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying semantic and acoustic tokens, Spee…

Go further

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

For now only among the papers summarized on this site: search again to add the prior work, the follow-ups and Semantic Scholar’s recommendations.

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

⚠The license of this specific paper does not allow us to build upon its content and the summarizing tools will be run using the paper metadata rather than the full article. However, it still does a good job, and you can also try our tools on papers with more open licenses.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.

Filler Word Detection and Classification: A Dataset and Benchmark

The essentials

Read further

Definitions

Introduction

The Need for Research on Filler Words

The PodcastFillers Dataset

The Proposed Pipeline

Results and Evaluation

Conclusion

Ask questions about this paper to our AI assistant

Tools for this paper

Advanced tools Premium

Papers around this one

Closest papers

Go further

Cite this paper (BibTeX)