Nowadays, Emotion Recognition (ER) is a crucial aspect of human–computer interaction which allows to understand and respond to human emotions. However, the existing Multimodal Fusion-based Deep Neural Network (MMF-DNN) failed in recognizing the emotions from a multimodality due to increased generalizability and scalability issues. Hence, this research proposes an effective Multimodal Transformer Fusion (MTF) which combines the strengths of various modalities to recognize human emotions. Initially, the data is collected from Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset and fed into preprocessing. The collected data is further preprocessed with Wavelet Threshold (WT), Min–max normalization, stop words’ removal, and lemmatization to remove noise and improve the quality of ER. Then, the features are extracted by using various techniques and fed into classification to classify the emotions into various categories. Finally, the proposed MTF is employed to integrate the classified features and provides a final prediction of recognized emotions. From the results, the proposed MTF achieved outstanding results in accuracy (Acc) of 93.61% when compared with the existing Multimodal ER with Audio-Visual Fusion (MER-AVF).