Themes of Research

　Gr. 2 promotes "MR-PreViz Project" for supporting filmmaking by using mixed reality (MR) technologies. In the preproduction stage of the filmmaking process, MR-PreViz system shows a cameraman CGI characters acting in open sets and locations through an HD video camera. Therefore, the cameraman and other crews can discuss, examine, and confirm what their film will look like. This project finished in 2010 and the research results are freely available.

Click here to "MR-PreViz Project website"

　In MR, merging real and virtual worlds requires for estimating camera pose accurately in real-time. This subject is called geometric registration, and our approach to the subject is categorized as vision based-method. Concretely, camera pose is estimated using LMDB in advance which stores geometric information of shooting scene measured using image information only. There are past studies that camera registration covers a wide range of shooting site by combining several LMDBs, and on-site LMDB updates by adopting SLAM algorithm.

"Camera pose estimation using LMDB"

　3D films became a big fad among film industry, and filmmakers desired for 3D version of MR-PreViz. To meet the expectation, we enabled MR-PreViz to be able to examine S3D effects on a screen by using a cinematographic stereo camera called stereo rig on which distance and angle between two cameras can be controlled. We named this new system S3D MR-PreViz. After the equipment and system update, we also focus on camera registration method using the rig.
　Previous work includes accuracy improvement of localization method using LMDB, and time-consuming preparation process for constructing the DB was removed. Currently, the main focus of this study is development of a novel registration method taking advantages of the rig.

"S3D MR-PreViz"

　There are various researches to mix virtual and real world without being felt uncomfortable in MR. In this case, it is important to conform to optical conditions such as shadows and image quality in MR Space. Furthermore, we engage in research for varying optical condition freely in MR space.
　Previous research, we have been studying "Relighting" that gives virtual light effects to both real and virtual objects and giving highlight and virtual reflection to real objects. Then, it is visually possible to represent a dark room, which is bright room in fact, and real object's material is virtually changed to another one.
　Moreover, we've studied how to measure reflective characteristic of real objects for realizing Relighting which belongs to reality. In addition, we've proposed the methods for acquiring HDRI (High Dynamic Range Image) which is able to represent changes of image quality and brightness close to human perception rather than original images'.

"Relighting"

　DR refers to technologies to visually diminishing, eliminating or seeing through obstacles in the real world. Compared to MR, which "adds" information, DR means "subtraction" (addition of negative information). It's because DR is accomplished by superimposing an area hidden from a user's view by the obstacles. In DR, which doesn't superimpose CG but an actual scene, even little gaps in image quality and in registration become too major problems to ignore. 　
　So, we try to accomplish DR perfectly from various approaches to these problems. Moreover, we also study representation methods "Half-DR" which left the obstacles slightly on purpose, in case of inconvenience of DR (e.g. user's collision with diminished objects).

"example of DR"

"Half-DR"

"Reduction of geometric inconsistency by using mosaicing"

　In this research, we focus on a gestural user interface which uses full -body motion interaction in 3D virtual space surrounding a user. We assume to use positions and orientations of multiple points on a user's body for determining the gestures. For that purpose, we are developing a software tool for aiding implementation of this kind of gesture interface into VR/MR application systems. Essential functions of the tool include assisting for gesture design, capturing and analyzing motion data, recognizing gestures. And we pursue the goal of systematizing gesture interface through development of application system with this tool.

"Gesture Design"