Order-aware Interactive Segmentation

Bin Wang1,2, Anwesa Choudhuri2, Meng Zheng2, Zhongpai Gao2, Benjamin Planche2, Andong Deng2,3, Qin Liu4, Terrence Chen2, Ulas Bagci1, Ziyan Wu2
1Northwestern University 2United Imaging Intelligence
3University of Central Florida 4University of North Carolina
arXiv Podcast Demo
Teaser Image

We propose OIS: order-aware interactive segmentation, to explicitly integrate missing relative depth information into 2D interactive segmentation.

Abstract

Interactive segmentation aims to accurately segment target objects with minimal user interactions. However, current methods often fail to accurately separate target objects from the background, due to a limited understanding of order, the relative depth between objects in a scene. To address this issue, we propose OIS: order-aware interactive segmentation, where we explicitly encode the relative depth between objects into order maps. We introduce a novel order-aware attention, where the order maps seamlessly guide the user interactions (in the form of clicks) to attend to the image features. We further present an object-aware attention module to incorporate a strong object-level understanding to better differentiate objects with similar order. Our approach allows both dense and sparse integration of user clicks, enhancing both accuracy and efficiency as compared to prior works. Experimental results demonstrate that OIS achieves state-of-the-art performance, improving mIoU after one click by 7.61 on the HQSeg44K dataset and 1.32 on the DAVIS dataset as compared to the previous state-of-the-art SegNext, while also doubling inference speed compared to current leading methods.

Methods


Each object has a specific order, or relative depth from each other, corresponding to its location in 3D space. By calculating the relative depth between each point in the image and the user prompt's location, we create order map, to effectively separate the prompt-selected object with others.

one click

(Red dots are positive clicks, blue dots are negative clicks, darker means closer to prompt-selected object, while lighter areas are farther to prompt-selected object)

Demo Videos

Gate

Moto

Four Consecutive Images

Elephant

Results


(1) Multi-clicks result comparison of SegNext (current SOTA) and Ours:

multi click

multi click


(2) One click result comparison of SegNext (current SOTA) and Ours:

one click

one click

one click

BibTeX

@misc{wang2024orderawareinteractivesegmentation,
      title={Order-Aware Interactive Segmentation}, 
      author={Bin Wang and Anwesa Choudhuri and Meng Zheng and Zhongpai Gao and Benjamin Planche and Andong Deng and Qin Liu and Terrence Chen and Ulas Bagci and Ziyan Wu},
      year={2024},
      eprint={2410.12214},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2410.12214}, 
}