Recently, Large Language Models (LLMs) like ChatGPT have become quite popular with the general public, enabling humans to chat with artificial agents. However, these LLMs do not understand human emotions as their responses often sound neutral and their goal is to support users in various capacities. In this work, we introduce Empathetic-HRI, a novel dataset comprising paired face images, contextual text, and emotion labels to facilitate multimodal empathetic response generation. In addition, we propose MATE, a multimodal agent that takes the human’s facial expressions and text as an input and responds back to the human in an empathetic manner. MATE is based on a transformer architecture, where the inputs are fed into a transformer encoder. The embeddings that are the output from the transformer encoder are then fed into the transformer decoder (Llama3). To evaluate the performance of our model, we conduct a survey and ask the participants if the responses given by the proposed model are appropriate and empathetic. The results are compared with ChatGPT and validated using a paired-samples T test. As the results are statistically significant, we prove that the responses given by MATE are both appropriate and empathetic.