PSG/SGG Dataset Debiasing

What is it about?

Panoptic Scene Graph Generation (PSG) translates visual scenes to structured linguistic descriptions, i.e., mapping visual instances to subjects/objects, and their relationships to predicates. However, the annotators’ preferences and semantic overlaps between pred- icates inevitably lead to the semantic mappings of multiple pred- icates to one relationship, i.e., biased-predicate annotations. As a result, with the contradictory mapping between visual and linguis- tics, PSG models are struggled to construct clear decision planes among predicates, so as to cause existing poor performances. Ob- viously, it is essential for the PSG task to tackle this multi-modal contradiction. Therefore, we propose a novel method that utilizes unbiased visual predicate representations for Biased-Annotation Identification (BAI) as a fundamental step for PSG/SGG tasks. Our BAI includes three main steps: predicate representation extraction, predicate representation debiasing, and biased-annotation identi- fication. With flexible biased annotation processing methods, our BAI can act as a fundamental step of dataset debiasing. Experi- mental results demonstrate that our proposed BAI has achieved state-of-the-art performance, which promotes the performance of benchmark models to various degrees with ingenious biased an- notation processing methods. Furthermore, our BAI shows great generalization and effectiveness on multiple datasets.

Featured Image

Photo by Gabin Vallet on Unsplash

Photo by Gabin Vallet on Unsplash

Why is it important?

Panoptic Scene Graph Generation (PSG) translates visual scenes to structured linguistic descriptions, i.e., mapping visual instances to subjects/objects, and their relationships to predicates. However, the annotators’ preferences and semantic overlaps between pred- icates inevitably lead to the semantic mappings of multiple pred- icates to one relationship, i.e., biased-predicate annotations. As a result, with the contradictory mapping between visual and linguis- tics, PSG models are struggled to construct clear decision planes among predicates, so as to cause existing poor performances. Ob- viously, it is essential for the PSG task to tackle this multi-modal contradiction. Therefore, we propose a novel method that utilizes unbiased visual predicate representations for Biased-Annotation Identification (BAI) as a fundamental step for PSG/SGG tasks. Our BAI includes three main steps: predicate representation extraction, predicate representation debiasing, and biased-annotation identi- fication. With flexible biased annotation processing methods, our BAI can act as a fundamental step of dataset debiasing. Experi- mental results demonstrate that our proposed BAI has achieved state-of-the-art performance, which promotes the performance of benchmark models to various degrees with ingenious biased an- notation processing methods. Furthermore, our BAI shows great generalization and effectiveness on multiple datasets.

Perspectives