Themes of Research

　DR refers to technologies to visually diminish, eliminate, or see through target objects from a video stream in real time. MR “adds” information generated by a computer to a real image, but DR subtracts information in a real image using image processing by a computer. Actually DR is accomplished by adding and overlaying “negative” image onto the real image, which is a reconstructed view hidden from a user by the target objects. Therefore, it is often the case that slight gaps between the real image and the overlaid image appear geometric and photometric inconsistency. The main research topic in Gr.2 is to minimize these inconsistencies using image processing.

　It is important for a study to prove the method is superior to the others. Of course, DR is no exception to this rule, but there is no appropriate datasets and methods for comparison. Therefore, we designed and built a studio for experimental data acquisition for evaluations and verifications in DR. In the studio, DR methods can be texted in various scenes under various lighting conditions. The studio consists of an indoor full-scale set and a 1/12 scale miniature set of an outdoor scene. Camera motion is controlled by an industrial 6DOF robot arm, and lighting conditions are changed by film quality lighting equipment. Manipulating these instruments, image sequences for DR evaluations are generated. Currently, we are searching for criteria of DR evaluations.

　As stated above, DR is accomplished by reconstructing and overlaying an image of a hidden view to a video stream. It is however difficult to reconstruct the view geometrically and photometrically correctly by Computer Vision, especially when the geometry is highly complex. As a result, the synthesized image appears as unnatural artifacts. To solve this problem, Light Field Rendering (LFR) is applied to DR to generate an arbitral viewpoint image using no geometric information. In LFR, pixels in a view are regarded as a bundle of light rays. This means that user’s viewpoint is reconstructed by collecting necessary light rays from a stored ray dataset. This approach is independent from scene geometry so suitable to solve above mentioned problems in DR.

　The aim of DR is to replace target objects to views hidden by the objects in a video stream, but it is also possible to represent the objects as semi-transparent on purpose. Against DR, we named and are studying such representation method Half-DR. Half-DR is useful if it is inconvenient to remove the target objects perfectly. For example, it can be used for collision avoidance after removing the objects by DR processing.

　Real-time camera pose estimation is one of the fundamental issues to seamlessly superimpose CGI in MR and hidden view in DR. This subject is known as geometric consistency, and, in Gr.2, vision-based approach is studied to solve this problem. In DR, it is often the case that 3D models of a hidden view with its color textures is used to fill in a target region. The main idea is to use these geometric and photometric data for camera pose estimation. Specifically, feature points are extracted from an image rendered using the 3D model and matched to the current image. Then, 3D positions of the feature points are extracted from a depth buffer. Finally, camera pose is calculated using the 2D-3D correspondences.

　Gr. 2 promotes "MR-PreViz Project" for supporting filmmaking by using mixed reality (MR) technologies. In the preproduction stage of the filmmaking process, MR-PreViz system shows a cameraman CGI characters acting in open sets and locations through an HD video camera. Therefore, the cameraman and other crews can discuss, examine, and confirm what their film will look like. This project finished in 2010 and the research results are freely available.