3D faces in motion: Fully automatic registration and statistical analysis

This paper presents a representation of 3D facial motion sequences that allows performing statistical analysis of 3D face shapes in motion. The resulting statistical analysis is applied to automatically generate realistic facial animations and to recognize dynamic facial expressions. To perform statistical analysis of 3D facial shapes in motion over different subjects and different motion sequences, a large database of motion sequences needs to be brought in full correspondence. Existing algorithms that compute correspondences between 3D facial motion sequences either require manual input or suffer from instabilities caused by drift. For large databases, algorithms that require manual interaction are not practical. We propose an approach to robustly compute correspondences between a large set of facial motion sequences in a fully automatic way using a multilinear model as statistical prior. In order to register the motion sequences, a good initialization is needed. We obtain this initialization by introducing a landmark prediction method for 3D motion sequences based on Markov Random Fields. Using this motion sequence registration, we find a compact representation of each motion sequence consisting of one vector of coefficients for identity and a high dimensional curve for expression. Based on this representation, we synthesize new motion sequences and perform expression recognition. We show experimentally that the obtained registration is of high quality, where 56% of all vertices are at distance at most 1 mm from the input data, and that our synthesized motion sequences look realistic.