Understanding Dataset Influence: Improving Machine Learning by Removing Harmful Data Samples

What is it about?

The study developed a data gradient framework for estimating the influence of individual training samples on machine learning model performance. This methodology calculates per-sample influence scores by combining sample-level gradients with inverse-Hessian-vector products, using LiSSA for large neural networks and conjugate gradient (CG) for smaller models. The approach embeds influence estimation into an iterative pipeline for dataset refinement, facilitating sample removal, label correction, and reweighting without offline processing. It introduces the dataset contribution score (DCS) to monitor dataset quality across refinement iterations. Empirical validation was performed across image, text, and tabular data using various model families, including neural networks and gradient boosted trees (GBTs). Experiments demonstrated that influence-guided dataset refinement improved classification accuracy and F1-scores compared to unrefined baselines. Scalability was confirmed with a training time overhead of approximately 0.44× for datasets up to 100,000 samples.

Featured Image

Why is it important?

This study is important as it addresses the critical challenge of improving machine learning system performance by enhancing training data quality. It introduces the data gradient framework, which efficiently quantifies the influence of individual samples on model performance, making it possible to refine datasets iteratively without computationally prohibitive costs. By integrating influence estimation into practical workflows, this research provides a scalable solution for systematically removing harmful samples and correcting labels, directly leading to improved model accuracy and reliability. This approach complements model-centric optimization and offers a significant advancement in data-centric machine learning workflows, driving better generalization and robustness across various data modalities and model architectures. Key Takeaways: 1. Iterative Dataset Refinement: The research highlights the effectiveness of using per-sample influence scores for iterative dataset refinement, resulting in classification accuracy improvements of 1.3%-2.3% and F1-score enhancements of 0.03-0.04 points over unrefined baselines. 2. Scalability and Efficiency: The study demonstrates that the data gradient framework maintains influence computation within a manageable 0.44× training time overhead for datasets up to 100,000 samples, ensuring practical applicability in large-scale machine learning tasks. 3. Robustness Across Modalities: Empirical validation across image, text, and tabular data shows that the framework significantly enhances model performance with effect sizes ranging from Cohen's d = 0.41 to 1.12, confirming the framework's versatility and effectiveness in diverse machine learning applications.