A Video Retrieval System with EVA-CLIP and Customized Keyframes Extraction

Abstract

Video retrieval is a crucial problem in computer vision and natural language processing, aiming to search and retrieve videos (or scenes, shots, and frames) based on a user given a video query. Among recent approaches for this task, keyframes-based content retrieval with a vision language model has shown significant efficiency and potential results. In this study, we evaluate various recent vision-language models for their effectiveness in replacing traditional CLIP in video retrieval problems with textual descriptions. We also introduced a customized keyframe extraction module using visual similarity to solve this issue. The findings indicate that, in terms of retrieval performance, EVA-CLIP and SigLIP notably surpass CLIP and alternative models when evaluated on the large-scale video datasets V3C. Furthermore, we introduce a customization keyframe extracting module to address the issue of information lacking in the default keyframe given in the dataset when compared to textual queries. Our customized keyframe extraction not only decreased the number of keyframes but also significantly improved retrieval outcomes compared to the default keyframes provided in the V3C dataset and extracted by Cineast.