A bi-modal face recognition framework integrating facial expression with facial appearance

Among many biometric characteristics, the facial biometric is considered to be the least intrusive technology that can be deployed in the real-world visual surveillance environment. However, in facial biometric, little research attention has been paid to facial expression changes. In fact, facial expression changes have often been treated as noise that would degrade the recognition performance. This paper studies an innovative viewpoint: (1) whether facial expression changes, namely facial behavior, can be positively used for face recognition or not? (2) furthermore, can facial behavior be integrated with facial appearance for assisting the extra-personal separation to enhance face recognition performance? We propose a bi-modal face recognition framework which integrates facial expression with facial appearance. Substantial experiments on multiple facial appearance and facial expression data have been conducted. Our experimental results have validated that facial behavior can play a positive role in face recognition and can assist facial appearance in extra-personal separation in multiple modalities for personal identification improvement.