2D Amodal Instance Segmentation Guided by 3D Shape Prior

Zhixuan Li1,2, Weining Ye2, Tingting Jiang1,2†, Tiejun Huang2
1Advanced Institute of Information Technology, Peking University, Hangzhou, China
2National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China

†Corresponding author.
ECCV 2022
Paper
Comparison between a 2D shape-prior dictionary and the proposed 3D shape-prior approach.

A finite 2D shape dictionary struggles with object viewpoints that were not observed during training. A3D reconstructs a complete 3D shape prior from each occluded instance and projects it to the estimated viewpoint, allowing the model to generate suitable 2D amodal masks adaptively.

Abstract

Amodal instance segmentation aims to predict the complete mask of an occluded instance, including both visible and invisible regions. Existing 2D methods learn complete silhouettes directly in image space, but these masks are only observations of a 3D model from particular viewpoints and therefore generalize poorly to unseen views. We build a bridge between occluded 2D instances and complete 3D models through reconstruction, using a 3D shape prior to guide 2D amodal segmentation. A3D reconstructs 3D models from occluded instances without requiring 3D annotations, then projects them into 2D masks under estimated viewpoints. Pretraining on large 3D reconstruction datasets handles shape diversity, and experiments show that A3D outperforms existing 2D amodal instance segmentation methods.

The Proposed A3D Approach

Overview of the Amodal 3D Network architecture.

A3D first predicts an amodal bounding box and visible mask. Its category-specific 3D modeling branch reconstructs a complete shape prior, while a viewpoint estimator predicts camera parameters. A differentiable renderer projects the reconstructed model into a coarse 2D amodal mask, which is then combined with image and visible-mask cues for final refinement.

Region-Specific Edge Refinement

The region-specific edge refinement module in A3D.

Mesh reconstruction captures the coarse complete shape but may lose fine boundaries. The region-specific edge refinement module combines the visible mask, cropped RGB appearance, and rendered coarse mask. It preserves reliable visible-region boundaries while refining the uncertain occluded region.

3D Shape Prior Reconstruction

Examples of reconstructed 3D shape priors and projected 2D amodal masks.

The reconstructed category-specific models retain complete object geometry even when the input is heavily occluded. Viewing each shape prior from multiple camera angles illustrates why 3D representation is more robust to viewpoint changes than memorizing a fixed collection of 2D masks.

BibTeX

@inproceedings{li2022a3d,
  title={2{D} Amodal Instance Segmentation Guided by 3{D} Shape Prior},
  author={Li, Zhixuan and Ye, Weining and Jiang, Tingting and Huang, Tiejun},
  booktitle={Proceedings of the IEEE/CVF European Conference on Computer Vision},
  pages={165--181},
  year={2022}
}