MERIT: Multimodal Enhanced Retrieval through Instruction-level Transformer-based Chunking in Product Manuals

Abstract

Product manuals are critical in helping users understand and operate consumer products. Yet, their increasing length and reliance on complex graphical elements often frustrate information retrieval. Traditional Product Manual Question Answering (PMQA) systems primarily focus on text and overlook visual content such as icons and diagrams, resulting in incomplete or ambiguous answers. This gap highlights the need for a multimodal approach that integrates textual and visual cues to ensure accurate and contextually grounded support. In this paper, we introduce MERIT, a novel multimodal QA framework that advances beyond page-level and text-only baselines. MERIT segments manuals into fine-grained, semantically coherent instruction chunks and encodes them using a Mixture-of-Experts Q-Former encoder, which effectively fuses textual summaries with graphical features. Trained under a contrastive learning objective, MERIT constructs a unified embedding space optimized for dense retrieval. A vision-language model then processes chunks and their associated graphics to generate accurate, fluent, and visually grounded answers. Extensive experiments on the PM209 dataset demonstrate that MERIT consistently improves retrieval precision and answer quality over state-of-the-art baselines, underscoring the importance of fine-grained multimodal reasoning for practical e-commerce applications.