Facial age recognition systems continue to be essential in several daily scenarios, particularly for minors’ security purposes. However, developing a robust and effective process remains challenging due to the unconstrained recognition environment of these systems. In fact, variation in facial expression during recognition can significantly decline system performance. Accordingly, the aging process varies according to gender, which makes accurate age recognition a challenging task. This paper presents a new approach for robust facial age recognition. The core idea of the proposed approach is to extract robust age features from emotion and gender characteristics. Once age representations are obtained, data augmentation is applied to enhance the generalization of the support vector machine classifier. An extensive experimental study was conducted to evaluate the proposed approach using the FG-NET, Adience, and CK+ datasets. The obtained performances demonstrate the effectiveness of our approach, which ranks top of the list of recent research with a 57% accuracy rate.