RUNE: Leveraging Neurosymbolic Inference For Remote Sensing Text-to-image Retrieval

What is it about?

This paper presents RUNE (Reasoning Using Neurosymbolic Entities), a neurosymbolic framework for text-to-image retrieval in remote sensing. RUNE combines large language models, object detection and segmentation, and symbolic reasoning to support accurate and explainable image retrieval. The system translates a user’s natural language query into a first-order logic representation, detects objects and their relationships in remote sensing images, and applies symbolic reasoning to evaluate the consistency between the entities and relations described in the query and those observed in the images. By explicitly reasoning over objects and their interactions, RUNE enables transparent retrieval decisions and significantly reduces false positives, providing a more reliable and interpretable alternative to purely neural approaches. As a use case to evaluate RUNE’s performance, we focus on post-flooding imagery, demonstrating its applicability to real-world scenarios.

Featured Image

Photo by Erik Mclean on Unsplash

Photo by Erik Mclean on Unsplash

Why is it important?

By employing symbolic reasoning for retrieval, our approach measures the compatibility between a natural language query and an image without relying on joint embedding generation. This design enhances explainability by making retrieval decisions directly traceable to the entities detected in the images and to those generated by the LLM during the transformation of the query into a first-order logic representation. In addition, symbolic reasoning significantly reduces false positives by enabling more precise control over reasoning and resource usage—an important advantage for deployment in real-world remote sensing scenarios. As a result, our methodology achieves more accurate and interpretable retrieval while maintaining execution times comparable to fully neural systems.

Perspectives