ACCV

VIPNet: Combining Viewpoint Information and Shape Priors for Instant Multi-View 3D Reconstruction

Weining Ye1, Zhixuan Li2, Tingting Jiang3

1Peking University

2Nanyang Technological University

3National Engineering Research Center of Visual Technology, Peking University

Paper BibTeX

Abstract

Overview

Existing multi-view reconstruction methods often fuse image features without explicitly using camera viewpoint, making it difficult to associate corresponding object parts. VIPNet introduces a viewpoint estimator that provides geometric context to each view and a shape-prior module that retrieves category knowledge when only a few observations are available. The resulting network reconstructs high-quality 3D point clouds in a single forward pass and generalizes to real-world images.

01 · Figure

VIPNet explicitly estimates camera viewpoints to associate features across views and retrieves shape priors to compensate for sparse observations, enabling fast and accurate 3D reconstruction.

Overview of the VIPNet architecture.
VIPNet explicitly estimates camera viewpoints to associate features across views and retrieves shape priors to compensate for sparse observations, enabling fast and accurate 3D reconstruction.

02 · Figure

Why Viewpoint Matters

VIPNet comparison with a baseline at different numbers of input views.
Viewpoint-aware association prevents geometrically distinct parts, such as the bow and stern of an airplane, from being confused during feature fusion.

03 · Figure

Shape-Prior Retrieval

Partial shape priors retrieved by VIPNet.
The shape-prior dictionary supplies plausible missing structure when the available views cover only part of the object.

04 · Figure

Real-World Generalization

VIPNet reconstructions from real-world multi-view images.
VIPNet transfers from synthetic training data to real photographs and reconstructs coherent object geometry from sparse views.

Citation

BibTeX

@inproceedings{ye2024vipnet,
  title={{VIPNet}: Combining Viewpoint Information and Shape Priors for Instant Multi-view 3D Reconstruction},
  author={Ye, Weining and Li, Zhixuan and Jiang, Tingting},
  booktitle={Proceedings of the Asian Conference on Computer Vision},
  pages={3379--3395},
  year={2024}
}