VIPNet: Combining Viewpoint Information and Shape Priors for Instant Multi-View 3D Reconstruction

Weining Ye1,Zhixuan Li2,Tingting Jiang3
1Peking University, 2Nanyang Technological University, 3National Engineering Research Center of Visual Technology, Peking University
Asian Conference on Computer Vision (ACCV), 2024
Paper
Overview of the VIPNet architecture.

VIPNet explicitly estimates camera viewpoints to associate features across views and retrieves shape priors to compensate for sparse observations, enabling fast and accurate 3D reconstruction.

Abstract

Existing multi-view reconstruction methods often fuse image features without explicitly using camera viewpoint, making it difficult to associate corresponding object parts. VIPNet introduces a viewpoint estimator that provides geometric context to each view and a shape-prior module that retrieves category knowledge when only a few observations are available. The resulting network reconstructs high-quality 3D point clouds in a single forward pass and generalizes to real-world images.

Why Viewpoint Matters

VIPNet comparison with a baseline at different numbers of input views.

Viewpoint-aware association prevents geometrically distinct parts, such as the bow and stern of an airplane, from being confused during feature fusion.

Shape-Prior Retrieval

Partial shape priors retrieved by VIPNet.

The shape-prior dictionary supplies plausible missing structure when the available views cover only part of the object.

Real-World Generalization

VIPNet reconstructions from real-world multi-view images.

VIPNet transfers from synthetic training data to real photographs and reconstructs coherent object geometry from sparse views.

BibTeX

@inproceedings{ye2024vipnet,
  title={{VIPNet}: Combining Viewpoint Information and Shape Priors for Instant Multi-view 3D Reconstruction},
  author={Ye, Weining and Li, Zhixuan and Jiang, Tingting},
  booktitle={Proceedings of the Asian Conference on Computer Vision},
  pages={3379--3395},
  year={2024}
}