Toward Robust Speech Emotion Recognition: A Hybrid EMix-Based Augmentation Approach

Abstract

Recently, various deep learning (DL) models have been introduced to enhance speech emotion recognition (SER) accuracy. However, the scarcity and limited scale of SER datasets due to the complexity and cost of data collection often result in model overfitting, thereby restricting overall performance. In this paper, we propose a hybrid data augmentation framework that integrates the previously introduced emotion-aware EMix method with complementary time-frequency perturbation techniques to enhance both model robustness and generalization. To validate the proposed approach, we develop a deep convolutional neural network consisting of two main components: a multi-branch stem block and a DenseNet-based backbone. The stem block extracts informative features across various time-frequency scales, while the backbone offers robust representational capacity based on a pre-trained image classification model. Experimental results on two publicly available benchmark datasets confirm the effectiveness of the proposed method, achieving state-of-the-art accuracies of 82.39% on CREMA-D and 78.25% on IEMOCAP.