face key point with 7 points

News

• 28-04-2015 - flandmark replaced by CLandmark!
• 11-11-2012 - New version of flandmark with better internal structure and improved MATLAB interface available!

Introduction

flandmark is an open source C library (with interface to MATLAB) implementing a facial landmark detector in static images. Detector parameters learning is written solely in MATLAB and is also part of flandmark.

The input of flandmark is an image of a face. Face detector provided by the courtesy of Eydea Recognition Ltd. was used to detect faces during the learning of parameters. However the flandmark software package includes fully-contained demo application which uses the OpenCV face detector.

flandmark (version 1.06) can be also used in python thanks to the following project: xbob.flandmark 1.0.2.

Sample results

The landmark detector processes each frame separately, i.e. temporal continuity of landmark positions is not exploited. The red rectangle is bounding box returned by the face detector, the blue rectangle represents the bounding box used to construct the input to flandmark detector (the normalized image frame).

Structured output classifier

flandmark uses a structured output classifier based on the Deformable Part Models (DPM). The quality of landmark configuration $\mathbf{s} = (\mathbf{s_0}, \dots, \mathbf{s_{M-1}}) \in \mathcal{S}$ for given image $I$ is measured by scoring function $f: \mathcal{I} \times \mathcal{S} \rightarrow \mathbb{R}$. The scoring function is defined as a sum of the appearance fit and the deformation cost. Exact formulation of $f$ is derived from the graph constraints:

Functions $q_i(I, \mathbf{s}_i)\ \mathrm{and}\ g_i(\mathbf{s}_i, \mathbf{s}_j)$ (appearance fit and deformation cost) are parameterized. These parameters are learned from annotated examples using the structured output SVM algorithm.

The maximization of $f$ is solved by Dynamic Programming (DP), thanks to the form of graph constraints (directed acyclic graph). The following images show the graph constraints for the current version of flandmark with 8 components, components dimensions and the flandmark detection pipeline.

Performance evaluaiton

The detector accuracy is measured in terms of the relative deviation defined as a distance between the estimated and the ground truth landmark positions divided by the size of the face. The size of the face is defined as the distance between the center of the mouth and the midpoint between centers of the eyes. The geometric accuracy is measured by the mean relative feature displacement and the maximum relative feature displacement.

$$ \mbox{mean relative displacement} = \frac{\epsilon_0+\epsilon_1+\cdots+\epsilon_7}{8} \cdot \frac{1}{l_{\mathrm{face}}} $$ $$ \mbox{maximum relative displacement} = \max \Bigg\{ \frac{\epsilon_0}{l_{\mathrm{face}}}, \cdots, \frac{\epsilon_7}{l_{\mathrm{face}}} \Bigg\} $$

The flandmark detector is compared with three competing detectors in terms of the accuracy of the estimated landmark positions. Specifically it is compared with detectors based on the following approaches: Active Appearance Models (AAM), the Deformable Part Models (DPM) based detector of Everingham, Sivic and Zisserman and binary SVMs trained independently for each landmark.

The parameters of flandmark detector were learned on a subset of the Labeled Faces in the Wild database (LFW). All performance evaluation was done on a subset of the LFW database.

Download

flandmark can be downloaded (or forked) from GitHub

Older versions:

• Version 1.07, 2012-11-11, flandmark_v107.zip (latest zip from GitHub). Downloads since 11-11-12: undefined
• Version 1.06, 2012-04-19, flandmark_v106.zip. Downloads since 19-04-12: undefined
• Version 1.05, 2012-03-27, flandmark_v105.zip (NOTE: WIP, version without learning scripts). Downloads since 29-03-12: undefined
• Version 1.04, 2012-03-01, flandmark_v104.tar
• Version 1.03, 2011-08-30, flandmark_v103.zip
• Version 1.02, 2011-08-10, flandmark_v102.zip

Annotated LFW database can be downloaded from here

• lfw.tgz, direct link to the LFW database on its homepage
• LFW_annotation.zip, LFW facial landmarks annotation (provided by courtesy of Eyedea Recognition Ltd.). Downloads since 29-03-12: undefined
• 08-06-12 LFW annotation updated, 15 images had wrong face bounding box assigned. We would like to thanks Alberto Albiol for noticing.

Code snippets

Usage of the flandmark detector in both C/C++ and MATLAB is very simple, as you can see from the following code snippets:

	  % Load image and convert it to grayscaleI = rgb2gray(imread('photo.jpg'));% Get face bounding boxbbox = [72, 72, 183, 183];% Load flandmark_model file into MATLAB memorymodel = flandmark_load_model('../data/flandmark_model.dat');% Detect facial landmark calling the mex functionlandmarks = flandmark_detector(I, int32(bbox), model);

	  #include "flandmark_detector.h"int main(int argc, char * argv[]){// load flandmark model structure and initializeFLANDMARK_Model * model = flandmark_init("flandmark_model.dat");// load input imageIplImage *img = cvLoadImage("photo.jpg");// convert image to grayscaleIplImage *img_grayscale = cvCreateImage(cvSize(img->width, img->height), IPL_DEPTH_8U, 1);cvCvtColor(img, img_grayscale, CV_BGR2GRAY);// bbox with detected face (format: top_left_col top_left_row bottom_right_col bottom_right_row)int bbox[] = {72, 72, 183, 183};// detect facial landmarks (output are x, y coordinates of detected landmarks)float * landmarks = (float*)malloc(2*model->data.options.M*sizeof(float));flandmark_detect(img_grayscale, bbox, model, landmarks);}

Platforms

GNU/Linux, Windows

flandmark should work with Mac also, though it was not tested.

Licensing information

FLANDMARK is licensed under the GNU/GPL version 3.

References

If you use this software in research, please cite this paper [Bibtex]

• M. Uricar, V. Franc and V. Hlavac, Detector of Facial Landmarks Learned by the Structured Output SVM, VISAPP '12: Proceedings of the 7th International Conference on Computer Vision Theory and Applications, 2012.Received Best Paper Award [pdf] [Bibtex]
• M. Uricar, Detector of facial landmarks, Master's Thesis, supervised by V. Franc, May 2011. [pdf] [Bibtex]
• J. Sivic, M. Everingham and A. Zisserman, "Who are you?" - Learning Person Specific Classifiers from Video, Proc. of IEEE Conference on Computer Vision and Pattern Recognition, 2009. [pdf]
• G. B. Huang, M. Ramesh, T. Berg and E. Learned-Miller, Labeled faces in the wild: A database for studying face recognition in unconstrained environments, Technical Report 07-49. University of Massachusetts, Amherst, 2007. [pdf]