Dual-Transformer Framework for Vietnamese Image-to-Poem Generation

Abstract

The Image-to-Poem is one of the novelty tasks in Artificial Intelligence and has received researchers' attention in recent years. This task requires the system to automatically generate a poem which is a creative content and specific structure in an aesthetically pleasing manner based on the input's image. However, most existing methods still have problems with topic inconsistency and irregularity to tackle this task. Moreover, the lack of benchmark datasets is a big problem because of the different points of view of a person which leads to difficulty in creating Image-to-Poem datasets, especially the low-resource languages. Therefore, in this paper, we present a Visual-68Poem dataset for the Vietnamese Image-to-Poem task with six-eight poems with a variety of content and context. In addition, we propose a Dual-Transformer architecture, including a component to extract the main objects and concept keywords from the image and a language model to generate a six-eight poem. Specifically, the generated poems must be similar to the context of the image and are complied with the rules of the structural genre of poetry. Experimental results on our standard dataset show that our proposed models consistently achieve competitive performance over other models on different measure scores. We release our dataset and code to facilitate future work on this task.