Automated Speech-To-Text Captioning for Videos and Noise Robustness Analysis Using OpenAI Whisper: A Performance and Enhancement Study

Abstract

Automatic speech recognition (ASR) based on deep learning has come a long way, thanks to the growing need for speech-to-text transcription in multimedia tools like video subtitling and accessibility tools. While OpenAI’s Whisper model has demonstrated state-of-the-art transcription performance across multiple languages and accents, its real-world deployment remains challenging due to linguistic variations, background noise, and formatting inconsistencies. This study investigates the application of Whisper for automated video captioning, evaluating its baseline transcription accuracy and exploring its limitations under diverse linguistic and environmental conditions. The research focuses on assessing Whisper’s weaknesses, particularly in accented speech recognition and robustness in noisy environments, and proposes a series of post-processing techniques to enhance its usability. The methodology consists of dataset pre-processing, model evaluation, real-world robustness testing, and the application of phonetic normalization, punctuation restoration, and spelling correction. The study result demonstrates that Whisper achieves a Word Error Rate (WER) of 4.75% after post-processing but struggles with Scottish-accented speech (WER 18.18%) and noisy environments. By introducing post-processing strategies, including phonetic adaptation and transcript enhancement techniques, the study significantly improves transcription accuracy, readability, and usability for real-world applications.