Traditional Retrieval-Augmented Generation (RAG) methods face significant limitations when dealing with visually rich documents. Existing research encounters two major bottlenecks: First, the evaluation baseline is limited to single-document or single-image scenarios, making it difficult to meet cross-document Q&A requirements. Second, mainstream VisualRAG methods are confined to page-image-level retrieval granularity, which leads to excessive redundant information in the retrieved results and makes it impossible to evaluate element-level retrieval effects. To address these issues, we construct MultiDocSeek, a large-scale cross-document Q&A benchmark dataset, whose multi-level retrieval labeling structure supports both page-image-level and element-level retrieval evaluation. Additionally, we propose the CoFi-VisRAG framework, which accurately matches information through a coarse-to-fine two-stage retrieval strategy: the first stage performs coarse-grained retrieval of page images, and the second stage retrieves fine-grained text and images within the page images. The experiments demonstrate that the framework significantly improves the accuracy of end-to-end Retrieval-Augmented Generation in both single-document and multi-document Q&A scenarios. MultiDocSeek example at https://github.com/liuchunming2/MultiDocSeek