LLMs Still Can't Avoid Instanceof: An Investigation Into GPT-3.5, GPT-4 and Bard's Capacity to Handle Object-Oriented Programming Assignments

Featured Image

Why is it important?

This work allowed us to understand the level of profiency that an LLM has while solving our OOP exercises. This was a fundamental step in reacting to the appearance of these new tools. We used the results from this study to perform changes to our OOP course, in order to prevent students from merely "copy-and-prompting" the assignments' instructions into the LLM and getting good grades without having built the relevant skills.