What is it about?
RepoMod-Bench evaluates AI coding agents on translating over 20 real-world repositories across 8 programming languages. The repositories range from a few thousand lines of code to hundreds of thousands. The benchmark relies on implementation-agnostic interfaces such as REST API or CLI to construct a hidden test suite that scores each translation on functional correctness, i.e. does the translated code actually behave the same as the original? Tests are hidden from agents so they cannot game the results. We evaluate four leading harness–model combinations and find a dramatic scaling collapse: pass rates drop from 91% on small projects to just 15% on large ones, revealing that autonomous code modernization at scale remains a significant open challenge.
Featured Image
Photo by Kevin Ku on Unsplash
Why is it important?
Existing code-generation benchmarks focus on function-level or file-level tasks, but real-world modernization projects involve translating entire repositories with thousands of interdependent files. There was no standardized way to evaluate agents on this task. RepoMod-Bench fills this gap with an evaluation framework that can be applied to any repository exposing a CLI or REST API, requiring no pre-existing spec beyond the source code itself. Our finding that agent architecture matters as much as the underlying model (a 6-point gap between harnesses using the same model) gives practitioners and researchers a concrete axis to optimize beyond simply scaling up model size. AI coding agents constantly increase the scope of work they can complete autonomously, to the point where it is unfeasible to manually review their output without causing a major bottleneck. This is exactly why automated evaluation methods are necessary - a good evaluation framework not only increases trust that the actual output matches the expected output, but can also serve as a feedback loop to the agent itself. RepoMod-Bench provides not just the dataset to measure different model-harness combinations, but the framework to easily extend the benchmark to additional repositories. This method can be used on a single repo when the goal is measuring the migration process and improving trust, but also on a large set of repos as a way to evaluate different models and harnesses for optimal performance in repo-level tasks.
Perspectives
At Modelcode we run large-scale code migrations and rewrites for enterprise customers every day, and we kept running into the same problem: how do you know the rewrite actually works without resorting to tedious manual review? This was our motivation behind this work. The scaling collapse we measured isn't surprising to anyone who has tried to migrate a large codebase with an AI agent, but now we have numbers to quantify exactly where agents break down and a public benchmark to track progress. We hope RepoMod-Bench becomes a useful tool for the community as agents get better at the kind of messy, large-scale work that matters most in practice.
Doron Serebro
Read the Original
This page is a summary of: RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing, August 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3770855.3817463.
You can read the full text:
Resources
Contributors
The following have contributed to this page
