### Comprehensive Analysis of Light Transport Operators for image synthesis (CaLiTrOp)

**Type of project:**ANR

**Contract dates:**2017 - 2022

**Équipe(s):**GeoMod, M2DisCo, Origami, R3AM

**Responsable scientifique LIRIS:**Nicolas Bonneel

**Partenaire(s):**INRIA Bordeaux - Sud-Ouest, INRIA Grenoble Rhône-Alpes, IRIT

**URL du projet:**https://www.irit.fr/calitrop/

**Description:**

Computing photorealistic images relies on the simulation of light transfer in a 3D scene, typically modeled using geometric primitives and a collection of reflectance properties that represent the way objects interact with light. Estimating the color of a pixel consists in integrating contributions from light paths connecting the light sources to the camera sensor at that pixel. Light paths are built as a combination of unitary light transport steps, such as reflectance, visibility, scattering, etc.
The space of light-paths, which dimension is infinite, is explicitly defined by the set of all possible light-paths built from these transport steps. Typical algorithms for off-line, high-quality images synthesis rely on the exploration, or sampling, of this space to select paths that are important for the computation of a picture. This exploration may be done deterministically or stochastically (for Monte-Carlo numerical integration method) and existing methods sample this space by leveraging on the local consistency that exists among light paths but hardly on the global structure, rather unknown, of this space.
A transversal view indeed, is to examine light transport operators from the point of view of infinite dimensional function spaces of light fields (imagine, e.g., reflectance as an operator that transforms a distribution of incident light into a distribution of
reflected light). This brings a totally new and interesting view on lighting simulation. Not only are these operators all linear in these spaces but they are also very sparse. As a side effect, the sub-spaces of light distributions that are actually relevant during the computation of a solution always boil down to a low dimensional manifold embedded in the full space of light distributions. Reflectance over ''smooth'' materials for instance, converts incident illumination into a low dimensional set of reflected light distributions.
The main questions thus concerns the identifying and the understanding of the global structure of the light-paths space, that is driven by the dimensionality and sparsity of light transport operators. What is the inherent dimensionality, topology and geometry of light-paths space? How can we leverage this information to improve lighting simulation algorithms? These are the questions that the project CALiTrOp wants to answer from a comprehensive functional analysis of light transport operators, with respect to the scene's geometry and the reflectance properties of the objects, but also, to link operators with screen-space visual effects, with respect to the resulting picture.
Studying the structure of high dimensional objects from a low dimensional set of observables is a problem that becomes ubiquitous nowadays: Compressive sensing, Gaussian processes, harmonic analysis and differential analysis, are typical examples of mathematical tools which will be of great relevance to study the light transport operators. Unsupervised machine learning from huge volume of generated data will also bring some helpful informations about these objects.
Expected results of the fundamental-research project CALiTrOp, whose deliverables will be scientific publications and technological demonstrators, are two-fold: (1) a theoretical understanding of the dimensionality and structure of light transport operators, bringing new efficient lighting simulation methods. Possible applications will cover re-lighting, material acquisition and efficient simulation of complex light transport problems (e.g. caustics). (2) We also envision that controlled approximations of the low dimensional operators will open the way to efficient approximations of light transport which may also be suitable for real time applications such as video games.
The expertise and the complementarity of the CALiTrOp academic consortium will allow to bring precise answers to the identified interrogations and to break the "local nature" wall of the state of the art in global illumination.