Scholarlink

This paper presents the development of a browser-based multimodal dialogue system that integrates speech recognition (STT), knowledge-based reasoning, and text-to-speech (TTS) synthesis, supported by Multimodal Large Language Models (MMLs). Speech is recorded via the browser using the MediaRecorder API and sent to a Flask backend, where Google Speech Recognition converts it to text. The recognized text is processed by an Ollama-based DeepSeek LLM, enhanced with a domain-specific knowledge base to generate appropriate responses. As a proof of concept, we developed a digital assistant called “OKGO Izakaya,” enabling users to engage in real-time multilingual conversations (English, Japanese, and Chinese) through the browser. The assistant recognizes spoken queries, reasons with the knowledge base, and responds with speech synthesized by gTTS or Microsoft Edge TTS. This implementation demonstrates the feasibility of lightweight, browser-compatible multimodal dialogue systems and provides a practical model for multilingual intelligent interactions.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Development of a Multimodal Dialogue System Integrating Speech Recognition, Knowledge Reasoning, and TTS via Multimodal Large Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Zhiyi Gao,
  • Akio Doi,
  • Toru Katoh,
  • Meguru Yamashita,
  • Hiroki Takahashi

摘要

This paper presents the development of a browser-based multimodal dialogue system that integrates speech recognition (STT), knowledge-based reasoning, and text-to-speech (TTS) synthesis, supported by Multimodal Large Language Models (MMLs). Speech is recorded via the browser using the MediaRecorder API and sent to a Flask backend, where Google Speech Recognition converts it to text. The recognized text is processed by an Ollama-based DeepSeek LLM, enhanced with a domain-specific knowledge base to generate appropriate responses. As a proof of concept, we developed a digital assistant called “OKGO Izakaya,” enabling users to engage in real-time multilingual conversations (English, Japanese, and Chinese) through the browser. The assistant recognizes spoken queries, reasons with the knowledge base, and responds with speech synthesized by gTTS or Microsoft Edge TTS. This implementation demonstrates the feasibility of lightweight, browser-compatible multimodal dialogue systems and provides a practical model for multilingual intelligent interactions.