This paper aims to make the reaching out process more comfortable and convenient for a person regarding their mental state as well as emotional state. This is achieved by developing a proposed system which is capable of recognizing mental health and emotion cues, facilitating a deeper understanding of human communication. The proposed system takes in the input of an exploratory questionnaire designed to understand the mental state and emotional state of the user as well as taking into consideration of the users’ comfort by addressing the questions in a more casual tone. It integrates multiple modalities, such as facial expressions and textual content to capture rich information and understand the context better by viewing the user’s reaction to the question. The facial along with text emotion recognition helps in understanding the emotional state while the response of the questionnaire can help in detecting depression and anxiety. We also determine additional features such as eye gaze while capturing facial emotion recognition to address linguistic ambiguity. With the help of evaluation metrics, the reliability of the proposed approach can be measured and identified. This paper enables the development of more empathetic and context-aware technologies, fostering improved interaction and engagement between humans and machines.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal System for Monitoring Mental and Emotional State with Deep Learning Model

  • V. Aishwarya,
  • D. Venkataraman,
  • M. C. Shunmuga Priya

摘要

This paper aims to make the reaching out process more comfortable and convenient for a person regarding their mental state as well as emotional state. This is achieved by developing a proposed system which is capable of recognizing mental health and emotion cues, facilitating a deeper understanding of human communication. The proposed system takes in the input of an exploratory questionnaire designed to understand the mental state and emotional state of the user as well as taking into consideration of the users’ comfort by addressing the questions in a more casual tone. It integrates multiple modalities, such as facial expressions and textual content to capture rich information and understand the context better by viewing the user’s reaction to the question. The facial along with text emotion recognition helps in understanding the emotional state while the response of the questionnaire can help in detecting depression and anxiety. We also determine additional features such as eye gaze while capturing facial emotion recognition to address linguistic ambiguity. With the help of evaluation metrics, the reliability of the proposed approach can be measured and identified. This paper enables the development of more empathetic and context-aware technologies, fostering improved interaction and engagement between humans and machines.