Investigating 3D Capsule Networks

Abstract

In recent years, convolutional neural networks (CNN) have been a key component in the progress of computer vision by means of weight sharing and translational invariance. However, there are two main downsides to CNNs: sub-sampling operation loses the precise spatial relationships, and CNNs cannot extrapolate their understanding of geometric relationships to entirely new viewpoints [1]. Sabour et al. proposed a novel neural network called capsule networks in 2017 in [2] to address these concerns. The capsule networks have achieved higher accuracy than traditional convolutional models with a similar number of parameters. Since its introduction, capsule networks have been used in various tasks in computer vision including shape alignment [3], video object segmentation [4], and classication of breast cancer [5].

In this research, we will explore two areas of improvement of capsule networks. Firstly, the interpretability of the existing capsule network architectures usually lack the ability to explain what each capsule corresponds to. Therefore, we will examine the latent capsule in the existing capsule networks to understand their meaning. Secondly, we will reconstruct 3D shapes from rendered images taken from different viewpoints. This path potentially includes implementing a previous work as the source code of this work has not been shared publicly.

Literature

[1] Geoffrey Hinton. Taking inverse graphics seriously.

[2] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Neurips, 2017.

[3] Yongheng Zhao, Tolga Birdal, Jan Eric Lenssen, Emanuele Menegatti, Leonidas Guibas, and Federico Tombari. Quaternion equivariant capsule networks for 3d point clouds. arXiv preprint arXiv:1912.12098, 2019.

[4] Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. Capsulevos: Semi-supervised video object segmentation using capsule routing. In ICCV 2019.

[5] Tomas Iesmantas and Robertas Alzbutas. Convolutional capsule network for classification of breast cancer histology images. In International Conference Image Analysis and Recognition, 2018.

[6] Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3d point capsule networks. In ICCV, 2019.