This paper explores the integration of a Vision-Language Model (VLM) and a cognitive architecture to create a context-aware, neuro-symbolic system capable of real-time media capture. The Learning Intelligent Decision Agent (LIDA), inspired by human cognition and the Global Workspace Theory, is paired with MobileCLIP, a state-of-the-art VLM that can interpret semantic information in visual data. This integration aims to enhance LIDA’s sensory system by providing robust, semantic understanding of real-world scenes while maintaining its decision-making and memory mechanisms. The system leverages MobileCLIP’s ability to process image-text embeddings for zero-shot activity recognition, and LIDA’s cognitive cycles enable adaptive behavior based on contextually relevant input. A proof-of-concept is demonstrated through a task where the system identifies and records specific actions during a track and field event. Several strategies for embedding comparison, including moving averages and cosine similarity, are evaluated for their effectiveness in accurately detecting the action. The results show that the integration of MobileCLIP with LIDA is promising, offering a scalable solution for real-time, context-aware decision-making. This work lays the foundation for future advancements in LIDA-based neuro-symbolic systems, particularly in environments requiring both perceptual understanding and adaptive cognitive functions. Future developments will focus on expanding the system’s capabilities to handle more complex tasks, integrating lifelong learning, and enhancing the robustness of decision-making processes.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Neuro-Symbolic LIDA’s Semantic Vision System

  • Nathan DiGilio,
  • Pulin Agrawal

摘要

This paper explores the integration of a Vision-Language Model (VLM) and a cognitive architecture to create a context-aware, neuro-symbolic system capable of real-time media capture. The Learning Intelligent Decision Agent (LIDA), inspired by human cognition and the Global Workspace Theory, is paired with MobileCLIP, a state-of-the-art VLM that can interpret semantic information in visual data. This integration aims to enhance LIDA’s sensory system by providing robust, semantic understanding of real-world scenes while maintaining its decision-making and memory mechanisms. The system leverages MobileCLIP’s ability to process image-text embeddings for zero-shot activity recognition, and LIDA’s cognitive cycles enable adaptive behavior based on contextually relevant input. A proof-of-concept is demonstrated through a task where the system identifies and records specific actions during a track and field event. Several strategies for embedding comparison, including moving averages and cosine similarity, are evaluated for their effectiveness in accurately detecting the action. The results show that the integration of MobileCLIP with LIDA is promising, offering a scalable solution for real-time, context-aware decision-making. This work lays the foundation for future advancements in LIDA-based neuro-symbolic systems, particularly in environments requiring both perceptual understanding and adaptive cognitive functions. Future developments will focus on expanding the system’s capabilities to handle more complex tasks, integrating lifelong learning, and enhancing the robustness of decision-making processes.