Semantic Morphable Models

In this thesis we discuss how computers can automatically interpret images of human faces. The applications of face image analysis systems range from image description, face analysis, interpretation, human-computer interaction, forensics to image manipulation. The analysis of faces in unconstrained scenes is a challenging task. Faces appear in images in a high variety of shape and texture and factors influencing the image formation process like illumination, 3D pose and the scene itself. A face is only a component of a scene and can be occluded by glasses or various other objects in front of the face.
We propose an attribute-based image description framework for the analysis of unconstrained face images. The core of the framework are copula Morphable Models to jointly model facial shape, color and attributes in a generative statistical way. A set of model parameters for a face image directly holds facial attributes as image description. We estimate the model parameters for a new image in an Analysis-by-Synthesis setting. In this process, we include a semantic segmentation of the target image into semantic regions to be targeted by their associated models. Different models compete to explain the image pixels. We focus on face image analysis and use a face, a beard and a non-face model to explain different parts of input images. This semantic Morphable Model framework leads to better face explanation since only pixels belonging to the face have to be explained by the face model. We include occlusions or beards as semantic regions and model them as separated classes in the implemented application of the proposed framework. A main cornerstone for the Analysis-by-Synthesis process is illumination estimation. Illumination dominates facial appearance and varies strongly in natural images. We explicitly estimate the illumination condition robust to occlusions and outliers.
This thesis combines copula Morphable Models, semantic model adaptation, image segmentation and robust illumination estimation which are necessary to build the overall semantic Morphable Model framework.