Configuring Test Generators using Bug Reports

What is it about?

The correctness of compilers is instrumental in the safety and reliability of other software systems, as bugs in compilers can produce executables that do not reflect the intent of programmers. Such errors are difficult to identify and debug. Random test program generators are commonly used in testing compilers, and they have been effective in uncovering bugs. However, the problem of guiding these test generators to produce test programs that are more likely to find bugs remains challenging. In this paper, we use the code snippets in the bug reports to guide the test generation. The main idea of this work is to extract insights from the bug reports about the language features that are more prone to inadequate implementation and using the insights to guide the test generators. We use the GCC C compiler to evaluate the effectiveness of this approach. In particular, we first cluster the test programs in the GCC bugs reports based on their features. We then use the centroids of the clusters to compute configurations for Csmith, a popular test generator for C compilers. We evaluated this approach on eight versions of GCC and found that our approach provides higher coverage and triggers more miscompilation failures than the state-of-the-art test generation techniques.

Featured Image

Photo by Alex Chumak on Unsplash

Photo by Alex Chumak on Unsplash

Why is it important?

Prior studies have proposed different techniques to exploit configurations to generate more effective test programs. To the best of our knowledge, the other comparable techniques do not use the information available in the bug reports and require a large initial test suite of failing and passing test programs generated by Csmith. Unfortunately, as GCC matures, it becomes harder for Csmith to trigger failures, which negatively impacts the applicability on newer, more stable versions of GCC. Instead, can we use insights from bug reports to guide Csmith? Analyzing bug reports can enable designing techniques to guide random test generators, such as Csmith, and ​can provide insights into the regions of code that are susceptible to bugs. To contrast with comparable techniques, we use real bugs from bug reports instead of using Csmith generated ones as the initial test suite and our approach also can work with few failing test programs. The evaluation results suggest that the configurations computed by our approach can help Csmith to trigger more miscompilation failures than the state-of-the-art approaches.

Perspectives