HyperNeRF

A Higher-Dimensional Representation
for Topologically Varying Neural Radiance Fields

Keunhong Park1,2
Utkarsh Sinha2
Peter Hedman2
Jonathan T. Barron2
Sofien Bouaziz2
Dan B Goldman2
Ricardo Martin-Brualla2
Steven M. Seitz1,2
1University of Washington, 2Google Research
Paper arXiv Code Data

HyperNeRF handles topological variations by modeling a family of shapes in a higher-dimensional space, thereby producing more realistic renderings and more accurate geometric reconstructions.

Here we show results generated with HyperNeRF. These videos show the input video being played back with a stabilized novel camera path. The right side video shows the depth of the scene. Click on the arrows or drag to see more results.

Abstract

Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous.

We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this "hyper-space". Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between "moments", i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS.

Video

Motivation

Level-Set Methods

HyperNeRF represents changes in scene topology by providing a NeRF with a higher-dimensional input. This is inspired by level-set methods. Level-set methods provide a means to model a family of topologically-varying shapes as slices of a higher dimensional auxiliary function. For example, these shapes

can be represented as slices through this auxiliary shape

We can naturally model topologically varying shapes by just moving the slicing plane along the higher dimensions. For example, this animation was generated by moving the slicing plane from top to bottom:

Slicing Surfaces

Consider the follow shapes, which have different permutations of O xand X.

Traditionally, level-set methods use straight planes to slice the higher-dimensional surface:

This means the higher-dimensional shape must contain copies of the same shape since each permutation has to lie along a single straight slice through the z-axis. If we let the slicing plane bend, it results in a much cleaner template:

Please see the paper for details.

HyperNeRF

The HyperNeRF architecture is a straightforward extension to Nerfies. The key difference is that the template NeRF is conditioned on additional higher-dimensional coordinates, where the higher dimensional coordinates are given by an "ambient slicing surface" which can be thought of as a higher dimensional analog to the deformation field.

HyperNeRF architecture.

Hyper-Space Template

HyperNeRF leverages main idea of level set methods by using a template NeRF which lives in higher dimensions. In addition to the spatial coordinates (X, Y, Z), the NeRF MLP takes additional higher dimensional coordinates W1 and W2. We call these the "ambient dimensions".

Here is an interactive viewer for the hyper-space of capture shown in the teaser. Drag the blue cursor around to change the ambient dimension rendered on the right.

Ambient Dimension Coordinates
(Background shows log density of coordinate)
The hyper-space template rendered from a fixed viewpoint.

BibTeX

@article{park2021hypernerf,
  author = {Park, Keunhong and Sinha, Utkarsh and Hedman, Peter and Barron, Jonathan T. and Bouaziz, Sofien and Goldman, Dan B and Martin-Brualla, Ricardo and Seitz, Steven M.},
  title = {HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields},
  journal = {ACM Trans. Graph.},
  issue_date = {December 2021},
  publisher = {ACM},
  volume = {40},
  number = {6},
  month = {dec},
  year = {2021},
  articleno = {238},
}

Acknowledgements

Special thanks to Aleksander Hołyński, Xuan Luo, and Haley Cho for their support and help with collecting data. Thanks to Zhengqi Li and Oliver Wang for their help with the NSFF experiments.