DaNangNLP Toolkit for Vietnamese Text Preprocessing and Word Segmentation

Abstract

Recent research has focused on Vietnamese large language models, however, the preprocessing steps play important complementary roles in the future success of Vietnamese language processing. In this paper, we design and develop a novel DaNangNLP toolkit that could cope with important Vietnamese language preprocessing steps. Although there have been many successful modules on Vietnamese language processing, existing toolkits still exhibit certain shortcomings, especially for word segmentation in complex Vietnamese sentences. Therefore, we have developed a practical and robust natural language processing pipeline specifically tailored for the Vietnamese language to address the challenging issues present in previous Vietnamese processing toolkits. The DaNangNLP pipeline based on the novel built-in word dictionaries is designed to handle Vietnamese text for typical preprocessing steps such as sentence segmentation, word regex, word normalization, and word segmentation. Throughout the evaluation, the proposed semantic-based word segmentation has outperformed the frequency-based word segmentation and existing toolkits in complex sentences.