What is it about?

Deep learning systems can make mistakes when they encounter data that differ from what they were trained on. Testing these systems can be costly because large amounts of test data may need to be manually labelled. Our study compares 15 methods for choosing the most useful test data. We examine how well they help find model errors, estimate model performance, and improve models through retraining across different types of data and distribution changes. Our findings provide practical guidance on which methods work best for different testing needs.

Featured Image

Why is it important?

It is important because deep learning systems can behave unexpectedly when they face new or unfamiliar data, which can be especially risky in applications such as autonomous driving or malware detection. At the same time, testing every possible input is expensive because many test cases require manual labelling. Our work helps practitioners choose a small but useful set of test data, making testing more efficient while still helping them uncover failures, assess model reliability, and improve models through retraining.

Perspectives

This publication is valuable because it moves beyond evaluating test selection methods in a single setting. By comparing many methods across different testing goals, data types, and distribution shifts, we provide a broader view of when these methods are actually useful. I hope the findings can help researchers design more reliable testing techniques and help practitioners choose suitable methods for evaluating deep learning systems in real-world environments.

Jingyu Zhang
Hong Kong Metropolitan University

Read the Original

This page is a summary of: Empirical Insights of Test Selection Metrics under Multiple Testing Objectives and Distribution Shifts, Proceedings of the ACM on Software Engineering, June 2026, ACM (Association for Computing Machinery),
DOI: 10.1145/3797086.
You can read the full text:

Read

Resources

Contributors

The following have contributed to this page