Thesis of Simon Forest

Topologies in neuro-inspired multimodal perception: learning, merging, decision-making

Defense date: 16/09/2022

Advisor: Salima Hassas
Coadvisor: Mathieu Lefort


This thesis focuses on multimodal merging on simulated topologies for use in an active perception context. As an example, humans receive dense information from multiple sensors and use various mechanisms to select and attend to only the relevant signals, for example by moving their eyes towards a target to see it better. Because of irregularities in sensory topologies (cf.\ fovea), actions can enhance perception, while extracting and merging data also helps choosing the best course of action. Similar needs are faced by artificial systems, e.g.\ social robots, albeit with their own set of physical constraints. This thesis proposes computational models for use in AI, taking inspiration from neuroscience case studies such as the superior colliculus, a subcortical structure involved in generating saccadic eye movements towards visual, auditory or multisensory stimuli.

When selecting information from multiple signals in a dynamic and multimodal setting, one needs a way to compute robust and reliable decisions. Decision-making in general has been tackled in either psychophysics or robotics using many different algorithms. One contribution of this thesis is to review and compare these algorithms, underlining their spatio-temporal properties, including feature merging, selective attention, etc. Among these models, dynamic neural fields (DNF) display some very interesting characteristics, including selective attention and data fusion depending on stimulus distance and precision. In another contribution, this thesis then makes use of DNF as a signal filtering and merging tool applied to multimodal fusion. This thesis shows how it can apply to model realistically occurences of the ventriloquist effect, a psychophysical effect of audio/visual stimulus localization capture. Then, in order to further study the role of topologies on these cognitive tasks, a final contribution shows that DNF retain their properties in irregular learned topological maps. In this experience, topologies are learnt via growing neural gas in order to extract intrinsic dimensions of the sensory space, but new perspectives, with deeper models, are suggested for application in active perception and embodied cognition.

Mme Estela BichoProfesseur(e)Centre Algoritmi, University of Minho, PortugalRapporteur(e)
Mme Salima HassasProfesseur(e)LIRIS, UCBLDirecteur(trice) de thèse
Mme Stéphanie Jean-DaubiasProfesseur(e)LIRIS, UCBLExaminateur​(trice)
M. Mathieu LefortMaître de conférenceLIRIS, UCBLCo-encadrant(e)
M. Jean-Charles QuintonMaître de conférenceLaboratoire Jean Kuntzmann, Univ. Grenoble AlpesCo-encadrant(e)
Mme Marieke RohdeDocteurInstitute for Innovation and Technology, GermanyExaminateur​(trice)
M. Denis SheynikhovichMaître de conférenceInstitut de la Vision, Sorbonne UniversitéRapporteur(e)
M. Jochen TrieschProfesseur(e)Frankfurt Institute for Advanced Studies, Goethe University, GermanyExaminateur​(trice)
M. Alban LaflaquièreDocteurSony AIInvité(e)