<p>Emotion recognition is the process of identifying human emotion. It is used in many fields such as psychology, medicine and security. Usually, any change in people’s emotions shows their appearance on the face. Findings have shown that changing emotions leads to changing the texture of the face image. In this paper, deep neural network is fed by texture information to extract pure combinational features. A deep convolutional neural network with three channels is designed in encoding format, where two channels are fed by facial texture features and one channel uses facial image information. Each channel is entered separately into an encoding structure of sequential convolutional layers and after generating the feature map. Next, a multi-layered architecture of fully connected layers is presented in decoding format, which is finally classified label of the image. An improved local ternary pattern descriptor has been used to extract image texture features. Experimental results showed that the proposed method provides 99.06%, 94.93% and 92.07% accuracy on JAFFE, KDEF and FER2013 datasets. The proposed approach provides higher classification accuracy than classical and deep-based compared methods on JAFFE, KDEF and FER2013 datasets. The presented solution for the simultaneous extraction of deep features along with handcrafted features can be used in many cases to classify visual patterns. Encoding–decoding format to extract deep concatenated feature-sets provide higher performance in comparing with classical sequential deep networks.</p>