Virtual reality (VR) refers to the set of technologies that allows a user to be transported into a world artificially generated by a computer -- also called virtual environment (VE). The user has the ability to interact in this environment, e.g., to move around, select and manipulate objects [1]. This ability to interact as well as the stimulation of several of the user's senses, allows one to be present in a VE as if it was a real environment. While most existing VR experiences are limited to audiovisual stimuli, recent surveys suggest going beyond this unique kind of stimulus. These notably show that the inclusion of auditory, olfactory, and tactile (including thermoception) stimuli seems to significantly enhance VR user experience, for several aspects [4,5]. A notable one is the benefits that come with increased degrees of presence and immersion for VR users. In addition, more and more affordable techniques have been recently developed to capture, create and render multisensory stimuli, whose digital representation is referred to as mulsemedia [2,3] -- for multiple sensorial media. These techniques enable the creation of what we may call multisensory digital twins. This concept extends the original digital twin concept beyond 3D graphics and data, by including sensory information (odors, tactile sensations, taste, sounds...). Such virtual multisensory replicas open the way to reproducing true sensitive experiences. Its application for sensory reconstitution of past scenes, for example, opens up very interesting perspectives for museography, and the preservation and transmission of tangible and intangible cultural heritage. This thesis focuses on the methodology and development of new techniques to record and immersively render multisensory stimuli, with a case study of multisensory digital twin creation for cultural heritage.
 

[1] Doug A Bowman, Ernst Kruijff, Joseph J LaViola, and Ivan Poupyrev. 2001. An introduction to 3-D user interface design. Presence 10, 1 (2001), 96–108.

[2] Gheorghita Ghinea, Christian Timmerer, Weisi Lin, and Stephen R. Gulliver. 2014. Mulsemedia: State of the Art, Perspectives, and Challenges. ACM Trans. Multimedia Comput. Commun. Appl. 11, 1s (oct 2014), 23–27.

[3] Alexandra Covaci, Longhao Zou, Irina Tal, Gabriel Miro Muntean, and Gheorghita Ghinea. 2018. Is multimedia multisensorial? - A review of mulsemedia systems. Comput. Surveys 51, 5 (2018), 23–27.

[4] Daniel Martin, Sandra Malpica, Diego Gutierrez, Belen Masia, and Ana Serrano. 2022. Multimodality in VR: A Survey. Comput. Surveys 54, 10s (sep 2022), 36.

[5] Miguel Melo, Guilherme Goncalves, Pedro Monteiro, Hugo Coelho, José Vasconcelos-Raposo, and Maximino Bessa. 2022. Do Multisensory Stimuli Benefit the Virtual Reality Experience? A Systematic Review. IEEE Transactions on Visualization and Computer Graphics 28, 2 (2022), 1428–1442.