SOHES: Self-supervised Open-world Hierarchical Entity Segmentation

Shengcao Cao1, Jiuxiang Gu2, Jason Kuen2, Hao Tan2, Ruiyi Zhang2,
Handong Zhao2, Ani Nenkova2, Liang-Yan Gui1, Tong Sun2, Yu-Xiong Wang1
1University of Illinois Urbana-Champaign, 2Adobe Research
arXiv OpenReview Video (YouTube) Video (Bilibili)

We present SOHES, a self-supervised method
for segmenting visual entities and their parts in an open world.

Refresh the page to see more examples.

Abstract

Open-world entity segmentation, as an emerging computer vision task, aims at segmenting entities in images without being restricted by pre-defined classes, offering impressive generalization capabilities on unseen images and concepts. Despite its promise, existing entity segmentation methods like Segment Anything Model (SAM) rely heavily on costly expert annotators.

This work presents Self-supervised Open-world Hierarchical Entity Segmentation (SOHES), a novel approach that eliminates the need for human annotations. SOHES operates in three phases: self-exploration, self-instruction, and self-correction. Given a pre-trained self-supervised representation, we produce abundant high-quality pseudo-labels through visual feature clustering. Then, we train a segmentation model on the pseudo-labels, and rectify the noises in pseudo-labels via a teacher-student mutual-learning procedure. Beyond segmenting entities, SOHES also captures their constituent parts, providing a hierarchical understanding of visual entities.

Using raw images as the sole training data, our method achieves unprecedented performance in self-supervised open-world segmentation, marking a significant milestone towards high-quality open-world entity segmentation in the absence of human-annotated masks.

Approach

Three phases of SOHES

Three phases of SOHES. In the first self-exploration phase, we cluster visual features from pre-trained DINO to generate initial pseudo-labels on unlabeled images. Then in the self-instruction phase, a segmentation model learns from the initial pseudo-labels. Finally, in the self-correction phase, we adopt a teacher-student framework to further refine the segmentation model.

Self-exploration phase for generating initial pseudo-labels

Self-exploration phase for generating initial pseudo-labels. This phase consists of four steps. We first merge image patches into regions with high visual feature similarities, then zoom in on the small candidate regions and re-cluster the local images to better discover small entities. After that, we refine the mask details and identify the hierarchical structure among the masks.

Results

Teaser

SOHES boosts open-world entity segmentation with self-supervision on various image datasets. Compared to prior state of the art, SOHES significantly reduces the gap between self-supervised methods and the supervised Segment Anything Model (SAM), using only 2% unlabeled image data as SAM.

Click on the buttons below to see the visual results on various image datasets,
and compare the inputs (top), SAM results (middle), and our SOHES results (bottom).

BibTeX

@inproceedings{cao2024sohes,
  title={{SOHES}: Self-supervised Open-world Hierarchical Entity Segmentation},
  author={Cao, Shengcao and Gu, Jiuxiang and Kuen, Jason and Tan, Hao and Zhang, Ruiyi and Zhao, Handong and Nenkova, Ani and Gui, Liang-Yan and Sun, Tong and Wang, Yu-Xiong},
  booktitle={ICLR},
  year={2024}
}