Sign Language Based Conversational Product Search
摘要
Though accessibility is one of the fundamental principles of inclusive design, the majority of product search systems are not sign language input-friendly, while sign language is an important means of communication for individuals with hearing or speaking disabilities. However, it is rarely used in search systems. In this work, we introduce a sign language translation module that strives to fill this gap by incorporating gesture-based interaction in DoodleShoper, a conversational product search assistant [5]. The system captures hand gestures using vision-based recognition [4] and converts them into textual output. These sentences are often incomplete or follow non-standard word order due to the structural nature of sign language. To address this, a large language model (ChatGPT) is employed to reconstruct coherent, contextually appropriate sentences by reordering and completing the recognized text. The refined result is then passed to DoodleShoper, enabling users to perform visual searches through conversational queries. This integration makes it possible for sign language users to access online visual information, such as products or related concepts, through gestures alone. By bridging gesture-based input with natural language and visual search, the system expands the usability of DoodleShoper and demonstrates a practical pathway for multimodal, AI-driven accessibility tools.