ECCV

2D Amodal Instance Segmentation Guided by 3D Shape Prior

Zhixuan Li1,2, Weining Ye2, Tingting Jiang1,2†, Tiejun Huang2

1Advanced Institute of Information Technology, Peking University, Hangzhou, China

2National Engineering Research Center of Visual Technology, School of Computer Science, Peking University, Beijing, China

†Corresponding author

Paper BibTeX

Abstract

Overview

Amodal instance segmentation aims to predict the complete mask of an occluded instance, including both visible and invisible regions. Existing 2D methods learn complete silhouettes directly in image space, but these masks are only observations of a 3D model from particular viewpoints and therefore generalize poorly to unseen views. We build a bridge between occluded 2D instances and complete 3D models through reconstruction, using a 3D shape prior to guide 2D amodal segmentation. A3D reconstructs 3D models from occluded instances without requiring 3D annotations, then projects them into 2D masks under estimated viewpoints. Pretraining on large 3D reconstruction datasets handles shape diversity, and experiments show that A3D outperforms existing 2D amodal instance segmentation methods.

01 · Figure

A finite 2D shape dictionary struggles with object viewpoints that were not observed during training. A3D reconstructs a complete 3D shape prior from each occluded instance and projects it to the estimated viewpoint, allowing the model to generate suitable 2D amodal masks adaptively.

Comparison between a 2D shape-prior dictionary and the proposed 3D shape-prior approach.
A finite 2D shape dictionary struggles with object viewpoints that were not observed during training. A3D reconstructs a complete 3D shape prior from each occluded instance and projects it to the estimated viewpoint, allowing the model to generate suitable 2D amodal masks adaptively.

02 · Figure

The Proposed A3D Approach

Overview of the Amodal 3D Network architecture.
A3D first predicts an amodal bounding box and visible mask. Its category-specific 3D modeling branch reconstructs a complete shape prior, while a viewpoint estimator predicts camera parameters. A differentiable renderer projects the reconstructed model into a coarse 2D amodal mask, which is then combined with image and visible-mask cues for final refinement.

03 · Figure

Region-Specific Edge Refinement

The region-specific edge refinement module in A3D.
Mesh reconstruction captures the coarse complete shape but may lose fine boundaries. The region-specific edge refinement module combines the visible mask, cropped RGB appearance, and rendered coarse mask. It preserves reliable visible-region boundaries while refining the uncertain occluded region.

04 · Figure

3D Shape Prior Reconstruction

Examples of reconstructed 3D shape priors and projected 2D amodal masks.
The reconstructed category-specific models retain complete object geometry even when the input is heavily occluded. Viewing each shape prior from multiple camera angles illustrates why 3D representation is more robust to viewpoint changes than memorizing a fixed collection of 2D masks.

Citation

BibTeX

@inproceedings{li2022a3d,
  title={2{D} Amodal Instance Segmentation Guided by 3{D} Shape Prior},
  author={Li, Zhixuan and Ye, Weining and Jiang, Tingting and Huang, Tiejun},
  booktitle={Proceedings of the IEEE/CVF European Conference on Computer Vision},
  pages={165--181},
  year={2022}
}