Panoptic Scene Graph Generation (PSG) is a recently introduced task aimed at solving problems related to image understanding, and it constitutes a significant segment of the computer vision area. Its objective is to detect relational triplets in images, namely, the subject, object, and their relationship, to summarize visual information in images succinctly. Compared to traditional Scene Graph Generation (SGG), PSG aims to utilize panoptic segmentation instead of rigid bounding boxes to generate scene graph representations, constructing more precise and comprehensive relationships through pixel-level segmentation. Currently, the performance metrics of PSG methods are low, making them difficult to apply in downstream tasks. Through our analysis, we have identified the breakthrough points for this issue. Initially, the relationships that need to be predicted are implicit between pairs of objects, thus, we argue that predicting relationships resembles detecting hidden information amongst paired objects. Furthermore, the relationship between two objects is not only related to them but also influenced by other objects, leading us to believe that an effective method requires a receptive field for global objects information.Based on these considerations, we propose adopting the DETR approach for predicting relationships in PSG. Transformers excel at forming global information, while pairing objects as input queries to transformer decoder allows for the anticipation of relationships implicit between these pairs. Experiments on the OpenPSG dataset resulted in R@20 and R@100 scores of 35.9% and 47.4%, respectively, marking a significant improvement over the baseline.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Relation Detection with Transformers for Panoptic Scene Graph Generation

  • Chang Liu,
  • Wenchao Yan,
  • Shilin Chen,
  • Liqun Huang,
  • Xiaotao Huang

摘要

Panoptic Scene Graph Generation (PSG) is a recently introduced task aimed at solving problems related to image understanding, and it constitutes a significant segment of the computer vision area. Its objective is to detect relational triplets in images, namely, the subject, object, and their relationship, to summarize visual information in images succinctly. Compared to traditional Scene Graph Generation (SGG), PSG aims to utilize panoptic segmentation instead of rigid bounding boxes to generate scene graph representations, constructing more precise and comprehensive relationships through pixel-level segmentation. Currently, the performance metrics of PSG methods are low, making them difficult to apply in downstream tasks. Through our analysis, we have identified the breakthrough points for this issue. Initially, the relationships that need to be predicted are implicit between pairs of objects, thus, we argue that predicting relationships resembles detecting hidden information amongst paired objects. Furthermore, the relationship between two objects is not only related to them but also influenced by other objects, leading us to believe that an effective method requires a receptive field for global objects information.Based on these considerations, we propose adopting the DETR approach for predicting relationships in PSG. Transformers excel at forming global information, while pairing objects as input queries to transformer decoder allows for the anticipation of relationships implicit between these pairs. Experiments on the OpenPSG dataset resulted in R@20 and R@100 scores of 35.9% and 47.4%, respectively, marking a significant improvement over the baseline.