Autonomous interaction with computers is a key opportunity for further advancement in the area of task automation, yet it has been an elusive goal for much of computer science history. The advances that have recently taken place within LLMs more than any other have sped up the advance of digital agents, though most are generally limited to relatively narrow domains, be it specific applications or web-based tasks. The approach overcomes the above limitation by introducing a versatile framework, OS-Copilot, for developing general-purpose agents that could interact with the OS's various elements, ranging from code terminals to file systems, multimedia, web applications, and third-party software. The OS-Copilot provides one single interface for the execution of tasks made of multiple methods of interaction, such as Python scripting, terminal commands, mouse and keyboard inputs, and API integrations. This holistic framework allows agents to interact fluently with the heterogeneous OS environment and does away with the imperative of having to design control mechanisms in a manual fashion for every application. The OS-Copilot, through self-learning, also allows agents to learn from experience and gather knowledge over previous tasks and improve their ability to work on unfamiliar applications without human assistance. This work, then, will be a basis for the development of general-purpose computer agents that can do a wide range of things and work under several OS environments-a situation that will finally get much closer to adaptive, self-improving digital assistants.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Advancements in Task Automation for Operating Systems: A Review of Language Agent Frameworks and Self-directed Learning

  • Shital Gajbhiye,
  • Nilam Jadhav,
  • Nihar Shettigar,
  • Sahil Omanwar,
  • Harsh Chavan,
  • Rahul Chauhan

摘要

Autonomous interaction with computers is a key opportunity for further advancement in the area of task automation, yet it has been an elusive goal for much of computer science history. The advances that have recently taken place within LLMs more than any other have sped up the advance of digital agents, though most are generally limited to relatively narrow domains, be it specific applications or web-based tasks. The approach overcomes the above limitation by introducing a versatile framework, OS-Copilot, for developing general-purpose agents that could interact with the OS's various elements, ranging from code terminals to file systems, multimedia, web applications, and third-party software. The OS-Copilot provides one single interface for the execution of tasks made of multiple methods of interaction, such as Python scripting, terminal commands, mouse and keyboard inputs, and API integrations. This holistic framework allows agents to interact fluently with the heterogeneous OS environment and does away with the imperative of having to design control mechanisms in a manual fashion for every application. The OS-Copilot, through self-learning, also allows agents to learn from experience and gather knowledge over previous tasks and improve their ability to work on unfamiliar applications without human assistance. This work, then, will be a basis for the development of general-purpose computer agents that can do a wide range of things and work under several OS environments-a situation that will finally get much closer to adaptive, self-improving digital assistants.