What is it about?
AI coding agents are often given step-by-step plans for fixing software bugs, but do they actually follow those plans? We investigated this question by analyzing 21,120 records of AI agents working on software repair tasks across four language models and different planning strategies. We found that providing a well-designed plan can help agents solve more problems, but agents do not always follow the instructions they receive. Poorly designed plans can even reduce their effectiveness, while periodic reminders can help them stay on track. Our research shows that understanding how AI agents work is just as important as measuring whether they successfully complete a task.
Featured Image
Photo by Volodymyr Hryshchenko on Unsplash
Why is it important?
As AI coding agents become increasingly capable of developing and maintaining software, it is important to understand whether they follow instructions reliably. Successfully fixing a bug does not necessarily mean an agent followed the intended process. Our study provides a systematic way to measure how closely AI agents follow prescribed plans and reveals that plan design can significantly affect their performance. Importantly, adding more instructions does not always lead to better results, and a poorly designed plan can be worse than having no plan at all. These findings can help researchers and developers design, evaluate, and train more reliable AI agents that follow useful instructions while adapting appropriately to complex tasks.
Perspectives
One of the most interesting findings of this work is that giving AI agents more detailed instructions does not necessarily improve their performance. Even seemingly useful steps can interfere with an agent's problem-solving process. I believe this raises an important question about how we evaluate AI agents. Rather than focusing only on whether they complete a task successfully, we should also examine how they arrive at their solutions and whether they follow the intended workflow. I hope this research encourages further investigation into building AI agents that can reliably follow instructions while remaining flexible enough to handle unfamiliar and challenging problems.
Saman Dehghan
University of Illinois at Urbana-Champaign
Read the Original
This page is a summary of: From Plan to Action: How Well Do Agents Follow the Plan?, October 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3832783.3834400.
You can read the full text:
Contributors
The following have contributed to this page
