Scholarlink

The integration of advanced artificial intelligence technologies in production lines enables an effective and safer collaboration between human workers and robotic platforms in Industry 5.0, where the focus is the reinforcement of the workers capability, improving their skills when cooperating with the robots. This novel collaboration paradigm requires a natural and efficient human-robot communication way in which speech command and gesture recognition emerge as fundamental components for enhancing collaboration and fostering adaptability in industrial environments. In this paper, we present an innovative multi-modal human-robot collaboration framework, based on speech command and gesture recognition, designed to meet the requirements of accuracy and real-time processing of an existing production line, used as test environment for the European FELICE project. The speech command recognition system performs voice activity detection and is able to reliably distinguish among a set of commands in a noisy industrial environments, by combining a Mel-spectrogram based representation for the voice with a speech recognition neural network based on a Conformer. As for the recognition of gestures, the task is performed using a one-stage detector based on MobileNetV3 SSD. The experiments, performed on datasets encompassing real, synthetic, and negative samples for speech commands as well as images acquired in a realistic use case for gesture recognition, have established the suitability of the proposed solution in challenging industrial settings.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi-modal Human-Robot Collaboration in Production Lines Through Speech Commands and Gestures

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Vincenzo Carletti,
  • Antonio Greco,
  • Domenico Longobardi,
  • Pierluigi Ritrovato,
  • Alessia Saggese,
  • Mario Vento

摘要

The integration of advanced artificial intelligence technologies in production lines enables an effective and safer collaboration between human workers and robotic platforms in Industry 5.0, where the focus is the reinforcement of the workers capability, improving their skills when cooperating with the robots. This novel collaboration paradigm requires a natural and efficient human-robot communication way in which speech command and gesture recognition emerge as fundamental components for enhancing collaboration and fostering adaptability in industrial environments. In this paper, we present an innovative multi-modal human-robot collaboration framework, based on speech command and gesture recognition, designed to meet the requirements of accuracy and real-time processing of an existing production line, used as test environment for the European FELICE project. The speech command recognition system performs voice activity detection and is able to reliably distinguish among a set of commands in a noisy industrial environments, by combining a Mel-spectrogram based representation for the voice with a speech recognition neural network based on a Conformer. As for the recognition of gestures, the task is performed using a one-stage detector based on MobileNetV3 SSD. The experiments, performed on datasets encompassing real, synthetic, and negative samples for speech commands as well as images acquired in a realistic use case for gesture recognition, have established the suitability of the proposed solution in challenging industrial settings.