Knowledge Negative Distillation: Breaking the Teacher’s Boundaries in Deepfake Detection

What is it about?

The rapid growth of deepfake technology has caused more and more problems from its misuse. Current detection systems often work well only on the data they were trained on, but fail when facing new kinds of forgeries. Our study finds that this happens because models become too focused on specific details in their training data and struggle to learn general patterns. To fix this, we introduce a new training method called Knowledge Negative Distillation (KND). It uses a teacher-student model setup, where the student learns not just to copy the teacher, but also to avoid repeating the teacher’s overfitting mistakes. We do this by encouraging the student to look beyond the teacher’s limited knowledge during training. We also design a flexible way to combine what both models learn, so that the final system can use the best parts of each. Experiments show that KND outperforms existing methods on multiple tests, offering stronger and more reliable deepfake detection. Because of its simple and general design, KND could also help improve performance in other areas where models suffer from overfitting.

Featured Image

Photo by Tom Kotov on Unsplash

Photo by Tom Kotov on Unsplash

Why is it important?

We reconsider the underlying causes of overfitting and identify that low-loss, overfitted features not only compromise adaptability to cross-domain data, but also limit the model’s capacity to acquire more broadly generalizable knowledge. Consequently, we posit that forgery detectors can significantly enhance their generalization performance by effectively bypassing these low-loss, overfitted features. Our proposed KND framework offers the following potential advantages: Universality: our model does not require any additional information except for RGB native data, which endows it with the potential to address a variety of cross-domain challenges across multiple modalities. Extensibility: remarkably, a lightweight student model has shown significant improvement in generalization performance compared to the teacher model.

Perspectives