Reinforcement learning for on-demand LLM retrieval and reasoning

What is it about?

This paper presents GraphRAG-R1, a novel GraphRAG framework enhanced with process-constrained reinforcement learning. By employing a modified GRPO algorithm along with Progressive Retrieval Attenuation (PRA) and Cost-Aware F1 (CAF) rewards, the method enables large language models to perform deep multi-hop reasoning. A three-stage training strategy and hybrid graph-textual retrieval mechanism allow the model to decompose complex queries, invoke retrieval tools on demand, and generate accurate answers efficiently.

Featured Image

Photo by Growtika on Unsplash

Photo by Growtika on Unsplash

Why is it important?

Existing GraphRAG methods often struggle with complex multi-hop reasoning due to rigid retrieval strategies and limited reasoning depth. GraphRAG-R1 is the first to introduce process-constrained reinforcement learning into GraphRAG, effectively addressing the challenges of shallow retrieval and overthinking. Experimental results show that it significantly outperforms state-of-the-art methods on both in-domain and out-of-domain datasets, and can be seamlessly integrated with various retrieval systems, advancing the reasoning capabilities of large language models in complex tasks.