Uncovering Bias in ASR Systems: Evaluating Wav2vec2 and Whisper for Dutch speakers

Marcio Fuckner; Sophie Horsman; Pascal Wiggers; Iskaj Janssen

Uncovering Bias in ASR Systems: Evaluating Wav2vec2 and Whisper for Dutch speakers

Marcio Fuckner, Sophie Horsman, Pascal Wiggers, Iskaj Janssen

Research output: Contribution to conference › Paper › Academic

433 Downloads (Pure)

Abstract

It is crucial that ASR systems can handle the wide range of variations in speech of speakers from different demographic groups, with different speaking styles, and of speakers with (dis)abilities. A potential quality-of-service harm arises when ASR systems do not perform equally well for everyone. ASR systems may exhibit bias against certain types of speech, such as non-native accents, different age groups and gender. In this study, we evaluate two widely-used neural network-based architectures: Wav2vec2 and Whisper on potential biases for Dutch speakers. We used the Dutch speech corpus JASMIN as a test set containing read and conversational speech in a human-machine interaction setting. The results reveal a significant bias against non-natives, children and elderly and some regional dialects. The ASR systems generally perform slightly better for women than for men.

Original language	English
Number of pages	6
Publication status	Published - 2023
Event	2023 International Conference on Speech Technology and Human-Computer Dialogue (SpeD) - National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania Duration: 25 Oct 2023 → 27 Oct 2023 https://sped.pub.ro/

Conference

Conference	2023 International Conference on Speech Technology and Human-Computer Dialogue (SpeD)
Abbreviated title	SpeD
Country/Territory	Romania
City	Bucharest
Period	25/10/23 → 27/10/23
Internet address	https://sped.pub.ro/

Access to Document

SpeD_Bias_ASR_PaperAuthor accepted manuscript (AAM), 308 KB

https://ieeexplore.ieee.org/document/10314895

Handle.net

Persistent link

Cite this

@conference{a911bf43476647fabd16b067d644af6e,

title = "Uncovering Bias in ASR Systems: Evaluating Wav2vec2 and Whisper for Dutch speakers",

abstract = "It is crucial that ASR systems can handle the wide range of variations in speech of speakers from different demographic groups, with different speaking styles, and of speakers with (dis)abilities. A potential quality-of-service harm arises when ASR systems do not perform equally well for everyone. ASR systems may exhibit bias against certain types of speech, such as non-native accents, different age groups and gender. In this study, we evaluate two widely-used neural network-based architectures: Wav2vec2 and Whisper on potential biases for Dutch speakers. We used the Dutch speech corpus JASMIN as a test set containing read and conversational speech in a human-machine interaction setting. The results reveal a significant bias against non-natives, children and elderly and some regional dialects. The ASR systems generally perform slightly better for women than for men.",

keywords = "speech recognition, Dutch, bias",

author = "Marcio Fuckner and Sophie Horsman and Pascal Wiggers and Iskaj Janssen",

year = "2023",

language = "English",

note = "2023 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), SpeD ; Conference date: 25-10-2023 Through 27-10-2023",

url = "https://sped.pub.ro/",

}

TY - CONF

T1 - Uncovering Bias in ASR Systems

T2 - 2023 International Conference on Speech Technology and Human-Computer Dialogue (SpeD)

AU - Fuckner, Marcio

AU - Horsman, Sophie

AU - Wiggers, Pascal

AU - Janssen, Iskaj

PY - 2023

Y1 - 2023

N2 - It is crucial that ASR systems can handle the wide range of variations in speech of speakers from different demographic groups, with different speaking styles, and of speakers with (dis)abilities. A potential quality-of-service harm arises when ASR systems do not perform equally well for everyone. ASR systems may exhibit bias against certain types of speech, such as non-native accents, different age groups and gender. In this study, we evaluate two widely-used neural network-based architectures: Wav2vec2 and Whisper on potential biases for Dutch speakers. We used the Dutch speech corpus JASMIN as a test set containing read and conversational speech in a human-machine interaction setting. The results reveal a significant bias against non-natives, children and elderly and some regional dialects. The ASR systems generally perform slightly better for women than for men.

AB - It is crucial that ASR systems can handle the wide range of variations in speech of speakers from different demographic groups, with different speaking styles, and of speakers with (dis)abilities. A potential quality-of-service harm arises when ASR systems do not perform equally well for everyone. ASR systems may exhibit bias against certain types of speech, such as non-native accents, different age groups and gender. In this study, we evaluate two widely-used neural network-based architectures: Wav2vec2 and Whisper on potential biases for Dutch speakers. We used the Dutch speech corpus JASMIN as a test set containing read and conversational speech in a human-machine interaction setting. The results reveal a significant bias against non-natives, children and elderly and some regional dialects. The ASR systems generally perform slightly better for women than for men.

KW - speech recognition

KW - Dutch

KW - bias

M3 - Paper

Y2 - 25 October 2023 through 27 October 2023

ER -

Uncovering Bias in ASR Systems: Evaluating Wav2vec2 and Whisper for Dutch speakers

Abstract

Conference

Access to Document

Handle.net

Fingerprint

Cite this