How People and AI Understand Code That Is Hard to Read

What is it about?

Our study compares how people and AI language models understand code that has been deliberately made harder to read without changing what it does. For example, a function named “add” might actually subtract two numbers. The misleading name does not change what the code does, but it may confuse the reader or change how they approach the code. We test changes to names and other features of code to ask whether the same changes confuse people and AI, or whether they struggle in different ways.

Featured Image

Photo by Markus Spiske on Unsplash

Photo by Markus Spiske on Unsplash

Why is it important?

We asked 50 computer science students and AI language models to predict what short programs would output across five versions of the code. Making code look harder did not always make it harder to understand. Students became more accurate when variables were renamed in Python, while their accuracy declined across the tested JavaScript versions. Among the models tested, only those trained specifically for reasoning tended to struggle on the same tasks as people. Daniel Kahneman’s distinction between quick impressions and careful reasoning suggests why misleading names might trick some readers but prompt others to slow down and check the instructions. In our study, correct answers to code with misleading names took longer than incorrect ones, consistent with this explanation. These findings suggest that code can change how people approach a problem, not simply make it harder. They also show why an AI model’s accuracy alone cannot tell us whether it struggles in the same situations as people. Understanding these differences could improve how we test AI coding tools and help programmers judge when AI offers a useful second opinion or might repeat their mistakes. It also opens questions about when encouraging AI to use more human-like strategies would help, and when different strategies would be more useful.

Perspectives