Reconstructing the 3D shape of objects from multiple images is an important goal in computer vision and has been extensively studied for both rigid and non-rigid (or deformable) objects. Structure-from-Motion (SfM) is an algorithm that performs the 3D reconstruction of rigid objects using the inter-image visual motion from multiple images obtained from a moving camera. SfM is a very accurate and stable solution. Deformable 3D reconstruction, however, has been widely studied for monocular images (obtained from a single camera) and still remains an open research problem. It involves an additional modelling of deformations which makes it rather complicated. Numerous strategies have been developed in the past 30 years, ranging from statistical approximations to physical ones (constraining lengths and other metric quantities), all missing a concrete, mathematical way of formulating deformations.

In our recent works[1,2,3], we have shown that breaking a deformable scene into infinitesimal planes allows the deformations to be considered as linear functions which leads to simple, fast solutions to their reconstruction through images. This brought a substantial improvement to the state of the art in terms of complexity, accuracy and robustness[3]. Furthermore, it provides a mathematical framework that combines various kinds of deformation modelling, ranging from isometry (paper-like objects) [1,3] to elasticity [2,4], using simple local structures that preserve length, area, angles and smoothness.

In this project, we will extend the local methods to bring them at par with their rigid counterparts. This involves reducing the computational complexity of image registration, depth computation and including multiple view constraints to simplify the deformation constraints [4]. We will explore the geometric formulations that can allow additional constraints to simplify the problem in order to achieve generic, robust and real-time solutions.

[1] Parashar et al, TPAMI 2017.Isometric Non-Rigid Shape-from-Motion with Riemannian Geometry in Linear Time.

[2] Parashar et al, TPAMI 2018. Local Deformable Reconstruction Using Cartan’s connections.

[3] Parashar et al, TPAMI 2021. Robust Isometric Non-Rigid Shape-from-Motion.

[4] Parashar et al, CVPR 2020. Local Non-Rigid Shape-from-Motion from Diffeomorphic Mappings.