Personality trait identification from social media data has many applications in health monitoring, education, candidate screening, and business. Personality trait identification using text and images published on Social media (Twitter) is challenging due to unpredictable text and the background in the personality trait images. Unlike the existing studies that use multimodal (image and text) for the classification of personality traits images, this study fuses the text of emotions, text in the personality traits image, and image information. We believe that there is a strong correlation between emotion and personality traits. This observation motivated us to combine the textual features of emotional images, text in personality traits images, and image features. The above observations are captured through transformer encoders and multiple convolutional layers, followed by max-pool layers. Before feeding to transformers, our method extracts text in the personality trait images, resulting in text regions and non-text regions separately. Then, the proposed model obtains captions for the emotion and personality trait images. To integrate the strengths of the features extracted from emotion and personality text and images, the proposed work introduces a bilinear fusion approach, which fuses features and modalities. The experiments are conducted on different standard datasets of personality trait images to demonstrate the effectiveness of classification. A comparative study with state-of-the-art methods shows that our method is superior to existing methods. Experiments are also conducted on different races and genders to validate bias and fairness.