HeadsUp：从多视角图像重建高质量3D高斯头部
Large-Scale High-Quality 3D Gaussian Head Reconstruction from Multi-View Captures
Apple 提出 HeadsUp，一种前馈方法，可从多相机设置重建高质量3D高斯头部。该方法采用编码器-解码器架构，将输入视图压缩为潜在表示，再解码为锚定在中性头部模板上的UV参数化3D高斯。模型在包含超过10,000名受试者的内部数据集上训练，达到SOTA重建质量，无需测试时优化即可泛化到新身份。
AuthorsEvangelos Ntavelis, Sean Wu†, Mohamad Shahbazi, Fabio Maninchedda, Dmitry Kostiaev, Artem Sevastopolsky, Vittorio Megaro, Trevor Phillips, Alejandro Blumentals, Shridhar Ravikumar, Mehak Gupta, Reinhard Knothe, Jeronimo Bayer, Matthias Vestner, Simon Schaefer, Thomas Etterlin, Christian Zimmermann, Mathias Deschler, Peter Kaufmann, Stefan Brugger, Sebastian Martin, Brian Amberg, Tom Runia
Copy Bibtex
We propose HeadsUp, a scalable feed-forward method for reconstructing high-quality 3D Gaussian heads from large-scale multi-camera setups. Our method employs an efficient encoder-decoder architecture that compresses input views into a compact latent representation. This latent representation is then decoded into a set of UV-parameterized 3D Gaussians anchored to a neutral head template. This UV representation decouples the number of 3D Gaussians from the number and resolution of input images, enabling training with many high-resolution input views. We train and evaluate our model on an internal dataset with more than 10,000 subjects, which is an order of magnitude larger than existing multi-view human head datasets. HeadsUp achieves state-of-the-art reconstruction quality and generalizes to novel identities without test-time optimization. We extensively analyze the scaling behavior of our model across identities, views, and model capacity, revealing practical insights for quality-compute trade-offs. Finally, we highlight the strength of our latent space by showcasing two downstream applications: generating novel 3D identities and animating the 3D heads with expression blendshapes.
† ETH Zurich
** Work done while at Apple
Figure 1: Overview of HeadsUp. Our method reconstructs high-fidelity 3D Gaussian heads from multi-view images. Given a set of input views, our model utilizes a transformer-based encoder and a 3D Gaussian decoder to predict UV-parameterized 3D Gaussians for both the foreground and background. The model is trained end-to-end using a combination of photometric and perceptual supervision.
Related readings and updates.
DSplats: 3D Generation by Denoising Splats-Based Multiview Diffusion Models
DSplats: 3D Generation by Denoising Splats-Based Multiview Diffusion Models
January 18, 2025research area Computer Vision
Generating high-quality 3D content requires models capable of learning robust distributions of complex scenes and the real-world objects within them. Recent Gaussian-based 3D reconstruction techniques have achieved impressive results in recovering high-fidelity 3D assets from sparse input images by predicting 3D Gaussians in a feed-forward manner. However, these techniques often lack the extensive priors and expressiveness offered by Diffusion…
HUGS: Human Gaussian Splats
HUGS: Human Gaussian Splats
December 7, 2023research area Computer Vision
Recent advances in neural rendering have improved both training and rendering times by orders of magnitude. While these methods demonstrate state-of-the-art quality and speed, they are designed for photogrammetry of static scenes and do not generalize well to freely moving humans in the environment. In this work, we introduce Human Gaussian Splats (HUGS) that represents an animatable human together with the scene using 3D Gaussian Splatting…
Discover opportunities in Machine Learning.
来源：Apple Machine Learning Research · machinelearning.apple.com