Recent advancements in emotion recognition have increasingly focused on multimodal approaches, which offer higher accuracy and robustness compared to traditional methods, which often rely on a single modality, such as facial expressions, vocal tone, or speech content etc.. Despite these improvements, several challenges remain. Individuals exhibit unique expressions of emotions, often utilizing different modalities in varying proportions. These proportions can also fluctuate over time, adding complexity to the task. Generalized models fail to capture these subtle, individualized expressions, highlighting the need for personalized models to enhance emotion recognition accuracy. In this work, we propose a novel ensemble model that leverages weighted averaging and iterative learning from user feedback to dynamically adjust the modality weights. Our methodology combines specialized models for facial expressions, vocal tone, and speech content, integrating their predictions through a weighted averaging ensemble. Moreover, weights are dynamically adjusted using an iterative learning process that incorporates explicit user feedback. This approach allows the model to capture user-specific modality weights, enabling more accurate predictions for the considered emotional states: happy, sad, angry, and neutral, compared to standard generalized ensemble models. By continuously learning and adapting to individual expressions, our model significantly improves the accuracy of emotion recognition in real-world scenarios.