Skin segmentation for robust face image analysis

This thesis presents novel techniques to address the challenge of outlier detection and removal in the context of face analysis from photographs. Given a face image, under arbitrary scene conditions, our goal is to automatically compute a binary map that indicates the locations of facial occlusions, such as hairstyle, beard, clothing or glasses, and other atypical elements. The motivation is that this information can help other face processing methods, which do not tackle this problem on their own, to improve their results with minimal algorithmic adjustments. The 3D Morphable Model is a good example for such a method, and serves as testbed for our finding.
Usually outliers are difficult to capture. By definition they represent unpredictable deviations from facial appearance, which elude a systematical analysis. The problem is, that outliers impair a face description by perturbing extracted features. This can lead to wrong classifications or otherwise defective outputs. Therefore, in the face recognition literature, several methods have been devised to deal with this phenomenon. However, these solutions are neither comparable to our approach, nor applicable to our target applications, as they are often suited to a specific feature representation and not comprehensive.
We address the outlier problem, for the first time, as a classical segmentation task. The main contribution of our work is an algorithm, which determines the location of outliers on a pixel scale, by partitioning a face image into skin and "non-skin" regions. The algorithm is designed to work completely automatic and, unlike conventional skin detection techniques, it does not depend on color input. The latter is accomplished by means of a novel low-level texture analysis procedure, which comprises an illumination compensation step and a subsequent matching of image regions with respect to a given sample of skin texture. The resulting texture features are segmented with a customized version of the supervised GrabCut method. In order to facilitate automation, we incorporate structural knowledge on faces from the 3D Morphable Model. It allows us to mark specific facial areas, which are utilized as skin samples as well as to inizialized the actual segmentation routine.
We demonstrate the significance of the skin segmentation on three applications. First, it serves as main component to create an outlier map, that works in combination with a slightly modified fitting algorithm, to greatly improve the visual quality of 3D Morphable Model reconstructions. The second application extends this capability and reuses the image content, associated with the outliers, to realize a high level photo manipulation, called Face Exchange. The aim here is to substitute faces between different images, without affecting the rest of the scene. The last contribution represents a novel approach to face recognition. We localize prominent irregularities in facial skin, particularly moles, in order to use their characteristic configuration within a face for identification. For this task the skin segments are of utmost importance, to ensure high detection accuracy, and expressiveness of the extracted features.