Scholarlink

<p>Multimodal large language models (MLLMs) are rapidly advancing tools capable of synthesizing text and image data, and their potential application with patient- and caregiver-generated health data is gaining increasing attention. Ecological momentary assessment (EMA), which captures real-time experiences using text, ratings, or photographs, offers valuable insights but produces large, diverse datasets that are challenging to integrate into routine clinical care. Photographs, in particular, provide unique contextual information and are currently underutilized due to the substantial time required for review. MLLMs can address this challenge by efficiently summarizing and interpreting mixed media EMA records. This pilot study evaluated GPT-4’s ability to generate accurate insights from EMA datasets containing photographs with and without accompanying text. Twelve health sciences students simulated the role of glioblastoma family caregivers and submitted photographs and text describing caregiving challenges and successes over seven days. GPT-4 produced photograph descriptions and interpretations and group themes, which were evaluated by expert raters and compared with participant narratives using sentiment and semantic similarity analyses. Results showed that GPT-4 was generally accurate in describing visual content but less reliable in capturing participant-intended meaning. Expert ratings indicated strong accuracy but moderate agreement with ground truth, and semantic similarity analysis revealed loose alignment between participant narratives and GPT-4 interpretations. GPT-4 also exhibited a tendency toward more positive sentiment ratings than participants. While most outputs were accurate, occasional misinterpretations highlighted potential safety concerns. Findings suggest that MLLMs show promise in analyzing caregiver-generated photographs but need continued refinement to maximize their utility within healthcare.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Harnessing multimodal large language models to interpret ecological momentary assessment-generated caregiving photographs

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月27日

查看全文
  • Christy Muasher-Kerwin,
  • M. Courtney Hughes,
  • Aida Sanatizadeh,
  • Jamie F. Mayer,
  • Hamed Alhoori

摘要

Multimodal large language models (MLLMs) are rapidly advancing tools capable of synthesizing text and image data, and their potential application with patient- and caregiver-generated health data is gaining increasing attention. Ecological momentary assessment (EMA), which captures real-time experiences using text, ratings, or photographs, offers valuable insights but produces large, diverse datasets that are challenging to integrate into routine clinical care. Photographs, in particular, provide unique contextual information and are currently underutilized due to the substantial time required for review. MLLMs can address this challenge by efficiently summarizing and interpreting mixed media EMA records. This pilot study evaluated GPT-4’s ability to generate accurate insights from EMA datasets containing photographs with and without accompanying text. Twelve health sciences students simulated the role of glioblastoma family caregivers and submitted photographs and text describing caregiving challenges and successes over seven days. GPT-4 produced photograph descriptions and interpretations and group themes, which were evaluated by expert raters and compared with participant narratives using sentiment and semantic similarity analyses. Results showed that GPT-4 was generally accurate in describing visual content but less reliable in capturing participant-intended meaning. Expert ratings indicated strong accuracy but moderate agreement with ground truth, and semantic similarity analysis revealed loose alignment between participant narratives and GPT-4 interpretations. GPT-4 also exhibited a tendency toward more positive sentiment ratings than participants. While most outputs were accurate, occasional misinterpretations highlighted potential safety concerns. Findings suggest that MLLMs show promise in analyzing caregiver-generated photographs but need continued refinement to maximize their utility within healthcare.