What is it about?
Modern AI systems can "read" documents by looking at them as pictures instead of extracting text — this lets them keep charts, tables, and layout intact. The catch is that this visual approach is slow and expensive: every page turns into a thousand tiny vectors, and searching a large document collection means comparing every query against every one of those vectors. This paper introduces a practical solution: instead of storing a thousand vectors per page, we group nearby image patches together and store just about thirty compact ones. Then we search in two quick passes — a fast one to shortlist candidates, and a precise one to rerank the top results. The technique works with any of the popular visual document models (ColPali, ColSmol, ColQwen2.5) without any retraining, extra data, or model changes. Across three standard benchmarks it makes retrieval about four times faster per query while keeping accuracy essentially unchanged at the ranks people actually look at (top 5–10 results). We released everything as an open-source Python package (visual-rag-toolkit) with a live demo, so anyone can try visual search on their own documents.
Featured Image
Photo by Igor Omilaev on Unsplash
Why is it important?
Visual document retrieval is one of the most exciting recent shifts in information retrieval: it lets AI systems work directly with the document as-published (charts, tables, layout, colour, non-Latin scripts) instead of a lossy text extraction. But it has been trapped in research demos because the compute and memory cost per query grows too fast to deploy at real scale. This work removes that barrier without asking anyone to retrain their model, fine-tune with new data, or wait for a new model release. The pooling recipes are a few dozen lines of NumPy; the two-stage retrieval runs server-side in Qdrant with a single API call. That means a small team can now stand up a production visual-search index over tens of thousands of documents on a single GPU - a workload that previously needed a cluster. The timing matters too: as VLMs like PaliGemma, Qwen2.5-VL, and SmolVLM become widely available, the bottleneck moves from "can we embed the page?" to "can we serve the search cheaply?" This paper answers the second question, and ships the code that lets others answer it for their own domain.
Perspectives
This work grew out of an engineering problem, not a research question. I was building a visual retrieval system on a real document corpus, and the early results with ColPali-style late interaction were genuinely exciting: the model kept charts, tables, and layout that OCR pipelines throw away, and it retrieved things text-only search simply could not find. But the scaling wall arrived almost immediately: every page produced roughly a thousand vectors, and both sides of the system suffered. Indexing new documents was slow enough that keeping the collection up to date felt impractical, and serving a single query on the growing index was expensive enough that going beyond a demo required hardware I did not have. Looking through the literature I found a lot of work on making embeddings narrower - Matryoshka-style dimensionality reduction, quantisation, product quantisation - but almost nothing on the other axis: making a multi-vector page use fewer vectors. That felt like a strange gap, because reducing the vector count is twice beneficial. It makes indexing cheaper, which matters every time the corpus grows, and it makes retrieval cheaper, which matters every time a user asks a question. Compressing dimensions helps only the retrieval half of that story. So I went the other way and asked the simpler question: how few vectors per page can I get away with, without touching the model? The answer turned out to be surprisingly few, once you respect the spatial structure of the page - group nearby patches into tiles, keep one vector per tile, and rely on a lightweight two-stage retriever to recover any precision you lose during shortlisting. The pooling recipe is a few dozen lines of NumPy, and both retrieval stages execute inside a single Qdrant call. What used to be a cluster workload became a single-GPU workload, with essentially the same top-10 quality as the uncompressed baseline. I released everything as `visual-rag-toolkit` - SDK, CLI, live demo — because the interesting part is that it should be trivial to apply to your own documents. I would love to see the community push the idea further: dynamic tile counts per page, learned tile boundaries, or combining vector-count compression with the dimensional tricks that already exist. There is still a lot of room between "raw MaxSim" and "one global vector."
Ara Yeroyan
American University of Armenia
Read the Original
This page is a summary of: Visual RAG at Scale: Tile-Level Spatial Pooling for Efficient Multi-Vector Document Retrieval, July 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3805712.3808383.
You can read the full text:
Contributors
The following have contributed to this page
