Visual Coherence

The characteristic of hybrid reality is that virtual objects are truly implanted into the real-time physical environment by integrating information from the physical environment, which is considered to be one of the key technologies to realize the fusion of human-computer vision. From the user's point of view, the realistic experience of mixed reality depends on the successful fusion of virtual and real images in the perceptual layer.

Any mixed reality system must provide a coherent set of sensory, especially visually, stimuli. Therefore, the research on how to improve the visual coherence (Visual Coherence) of the picture and eliminate the discontinuous visual experience in the mixed reality environment is a hot issue to be solved in the field of graphic display.

Visual Coherence of Illumination

For a long time, the research on visual coherence usually focuses on geometric occlusion and physical collision, but does not adjust the lighting effect of virtual objects. No shadows are generated between virtual and real objects, and no lighting shading occurs. This mixed reality system does not produce a high level of reality, and the consistency between virtual and real environments is limited to geometry. Hybrid reality lacks the lighting information in the real scene in the process of rendering the virtual scene, which results in a gap between the lighting effect (diffuse reflection, specular reflection, shadow, etc.) of the rendered virtual object and the physical lighting environment. The body will produce significant perceptual differences between the virtual image and the real scene.

MR Real-time Rendering

Based on the consideration of illumination visual coherence, mixed reality rendering must support virtual lighting rendering of real objects (Re-Lighting). On the other hand, it also needs to support physical light source recognition (Light Estimation) to support physical lighting rendering of virtual objects. As shown in the figure on the left, it is feasible to render the graphics of different layer separately and display them through the head-mounted display of VST/OST.

Considering the computing power of wearable devices, this work is bound to use distributed computing and lightweight models. On this basis, the author proposes a real-time rendering / model training optimization method based on prior knowledge.

a Priori Perception

A large number of research results have shown that human visual experience can affect the effect of visual perception to a certain extent, and these prior knowledge can form a memory representation in the brain, thus affecting the result of visual perception (psychological image). This kind of perceptual a priori (visual illusion) can be applied to real-time rendering and machine learning. Reduce the operation pressure of rendering or training from an empirical point of view.

A priori in brightness and color perception

A priori perception of light direction

Optimization of Real-time Rendering

For example, in rendering, by analyzing the visual characteristics of the real light source, the author proposes to add light source details to the self-illuminating object through post-processing, which avoids the rendering of indirect lighting lampshade and brings a better rendering effect. At the same time, we use curtain and Yeelight intelligent lamps to build a light source controllable indoor environment, match and measure the light source rendering strategy in the mixed real environment, and map the physical parameters and rendering parameters to a certain extent.

Darkroom and Controllable Light Source

Optimization of machine learning

At present, the author is making a synthetic data set. By using semantic segmentation images with depth, normal and illumination information in the training of Discriminator network, the idea of auxiliary learning can effectively improve the accuracy of heavy illumination GAN (this idea is being verified at present).

Contact

Jiulong Lake Campus,Southeast University,Nan-Jing,CHINA
Phone : +86 176 025 34113
Email : esukastudio@outlook.com

Copyright © 2022.Esuka Studio All rights reserved.