Edge Vision Agents: Democratizing Computer Vision with LLMs
摘要
This paper presents an innovative approach to computer vision for edge devices through the integration of Large Language Models. A system is introduced that enables natural language control of computer vision operations on Raspberry Pi devices, effectively bridging the gap between human intent and technical implementation. The system architecture combines a central server running the Mistral-small [2] model with edge devices executing vision tasks through a specialized agent interface. The system implements a two-phase execution strategy: a planning phase for task interpretation and an implementation phase for execution, achieving 95% accuracy in command interpretation while maintaining robust vision processing performance. Through extensive experimentation with 1500 diverse images from the Common Objects in Context (COCO) [1] dataset, the system’s capability is demonstrated to handle tasks ranging from basic image processing to advanced computer vision operations. Performance analysis reveals average processing times of 0.009 s for fundamental operations and 556.2 ms for complex vision tasks, with real-time segmentation. The system excels in both single-operation commands and complex multi-step instructions, demonstrating 98.5% accuracy for basic tasks and 92.3% for complex workflows. The implementation significantly reduces the expertise barrier for computer vision applications while maintaining performance standards suitable for practical deployment. The results demonstrate the viability of natural language interfaces for edge-based computer vision, with particular promise in educational, industrial, and research applications. This work contributes to the democratization of computer vision technology by providing an intuitive interface for sophisticated image processing operations on resource-constrained devices.