Reasoning Scene Geometry from Single Images

Abstract

Holistic scene understanding is one of the major goals in recent research of computer vision. Most popular recognition algorithms focus on semantic understanding and are incapable of providing the global depth information of the scene structure from the 2D projection of the world. Yet it is obvious that recovery of scene surface layout could be used to help many practical 3D-based applications, including 2D-to-3D movie re-production, robotic navigation, view synthesis, etc. Therefore, we identify scene geometric reasoning as the key problem of scene understanding. This PhD work makes a contribution to the reconstruction problem of 3D shape of scenes from monocular images. We propose an approach to recognise and reconstruct the geometric structure of the scene from a single image. We have investigated several typical scene geometries and built a few corresponding reference models in a hierarchical order for scene representation. The framework is set up based on the analysis of image statistical features and scene geometric features. Correlation is introduced to theoretically integrate these two types of features. Firstly, an image is categorized into one of the reference geometric models using the spatial pattern classi cation. Then, we estimate the depth pro le of the speci c scene by proposing an algorithm for adaptive automatic scene reconstruction. This algorithm employs speci cally developed reconstruction approaches for di erent geometric models. The theory and algorithms are instantiated in a system for the scene classi cation and visualization. The system is able to fi nd the best fi t model for most of the images from several benchmark datasets. Our experiments show that un-calibrated low-quality monocular images could be e fficiently and realistically reconstructed in simulated 3D space. By our approach, computers could interpret a single still image as its underlying geometry straightforwardly, avoiding usual object occlusion, semantic overlapping and defi ciency problems.