Extracting procedural workflows from instructional videos is an emerging challenge at the intersection of video understanding, human activity recognition (HAR), screen-based task modeling, and domain-specific training, such as cybersecurity. Unlike conventional HAR, which focuses on physical actions, procedural extraction requires symbolic reasoning across multimodal inputs—command-line interfaces, GUIs, and narration. Current video-language models like struggle in these contexts due to limited symbolic grounding, weak task hierarchy modeling, and insufficient domain-specific data. To address this, we define the Digital Activity Recognition (DAR) problem class and implement it through the DAR Research Infrastructure Layer (DRIL). DRIL is validated on curated cybersecurity training videos, supporting DAR as a foundational task in multimodal AI through a reusable infrastructure for systematic experimentation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Digital Activity Recognition and Cybersecurity Procedural Task Extraction with Video LLMs: A Cognitive Computing Framework

  • Terry Traylor,
  • Ben Bernard,
  • Pann Ajjimaporn

摘要

Extracting procedural workflows from instructional videos is an emerging challenge at the intersection of video understanding, human activity recognition (HAR), screen-based task modeling, and domain-specific training, such as cybersecurity. Unlike conventional HAR, which focuses on physical actions, procedural extraction requires symbolic reasoning across multimodal inputs—command-line interfaces, GUIs, and narration. Current video-language models like struggle in these contexts due to limited symbolic grounding, weak task hierarchy modeling, and insufficient domain-specific data. To address this, we define the Digital Activity Recognition (DAR) problem class and implement it through the DAR Research Infrastructure Layer (DRIL). DRIL is validated on curated cybersecurity training videos, supporting DAR as a foundational task in multimodal AI through a reusable infrastructure for systematic experimentation.