Scholarlink

Screen readers present one cell of a spreadsheet at a time and provide only a limited overview on a single table such as its dimensions. Screen reader users struggle, for example, in recognizing labels to explain another cell’s purpose. In a Wizard-of-Oz study with two wizards as voice agents generating speech feedback we explore a novel audio-tactile graphics reader with tactile grid-based overlays and spoken feedback for touch to enable screen reader users to engage in a conversation about the spreadsheet with a voice assistant and utilize a screen reader to solve spreadsheet calculation tasks. In a pilot with 3 and a main study with 8 BLV students, we identify multi-modal interaction patterns and confirm the importance of two separate roles of speakers: a voice assistant and a screen reader. The conversation is driven by user’s multi-modal speech input and hand gestures provided sequentially or in parallel. Verbal references to cells by spoken addresses, values, and formulas can be embodied as tangible objects to unify tactile and verbal representations.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Integrating Touch, Gestures and Speech for Multi-modal Conversations with an Audio-Tactile Graphics Reader

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Bela Usabaev,
  • Nicola Bührich,
  • Tutku Can Taş,
  • Gerhard Weber,
  • Petra Ondrusek

摘要

Screen readers present one cell of a spreadsheet at a time and provide only a limited overview on a single table such as its dimensions. Screen reader users struggle, for example, in recognizing labels to explain another cell’s purpose. In a Wizard-of-Oz study with two wizards as voice agents generating speech feedback we explore a novel audio-tactile graphics reader with tactile grid-based overlays and spoken feedback for touch to enable screen reader users to engage in a conversation about the spreadsheet with a voice assistant and utilize a screen reader to solve spreadsheet calculation tasks. In a pilot with 3 and a main study with 8 BLV students, we identify multi-modal interaction patterns and confirm the importance of two separate roles of speakers: a voice assistant and a screen reader. The conversation is driven by user’s multi-modal speech input and hand gestures provided sequentially or in parallel. Verbal references to cells by spoken addresses, values, and formulas can be embodied as tangible objects to unify tactile and verbal representations.