Optimization of non-smooth functions via differentiable surrogates

What is it about?

This research paper presents a novel approach to optimize non-differentiable machine learning models by using differentiable surrogate models. The authors address a fundamental challenge in optimization: while modern machine learning algorithms like XGBoost and random forests excel at prediction accuracy, they are non-differentiable or discontinuous, making traditional gradient-based optimization methods inapplicable. The proposed method combines two models: XGBoost for accurate predictions and neural networks as differentiable surrogates to provide gradient information. During optimization, the neural network's gradients guide the SLSQP (Sequential Least Squares Programming) optimizer to find optimal solutions, while the final predictions are obtained using XGBoost's superior accuracy. The researchers tested their approach on three classical benchmark functions (Rosenbrock, Levy, and Rastrigin) with varying dimensions and constraint conditions, as well as a real-world steel alloy optimization problem. The experiments consistently demonstrated that the differentiable surrogate model achieved solutions up to 40% better than traditional methods while reducing computation time by orders of magnitude. The method maintained near-zero constraint violations across all test cases, even as problem complexity increased. The framework offers a practical solution for optimizing non-differentiable machine learning models and can be extended to other tree-based ensemble algorithms like LightGBM and random forests, providing flexibility while preserving optimization advantages.

Featured Image

Photo by Benjamin Lizardo on Unsplash

Photo by Benjamin Lizardo on Unsplash

Why is it important?

Bridges a critical gap: It resolves the fundamental tension between model accuracy (non-differentiable models like XGBoost) and optimization efficiency (requiring gradients) Practical applications: Enables efficient optimization in real-world engineering and industrial applications where accurate predictions and optimal solutions are both crucial Computational efficiency: Dramatically reduces optimization time from minutes to seconds while improving solution quality Constraint handling: Maintains excellent constraint satisfaction, critical for real-world applications with strict physical or regulatory requirements Broad applicability: The methodology can be extended to various tree-based ensemble algorithms, making it widely useful across different domains

Perspectives