Fourier Neural Networks for 3D Scene Understanding

Abstract

In the scope of this guided research, we want to investigate the effect of changing dimensions and domain of data with Fourier transform. We will stress some existing approaches and try to get their potentials. So far, a couple of works in the literature showed that manipulating the input embedding, or using sinusoidal activation functions were successful in modeling implicit functions (eg. NeRF^? [1], SIREN [2]), backed up by a theory that these networks approximate Neural Tangent Kernels. More recently, a generalized Fourier neural operator was introduced by Zongyi et al. [3] which demonstrated that such networks were indeed successful in solving higher order partial differential equations.

In the scope of this research, we aim to compare these different approaches on first a toy task, ie. depth regression, then on 3D view synthesis task for indoor domain.

Literature

[1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934, 2020.

[2] Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint arXiv:2006.09661, 2020.

[3] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.