Comparative Analysis of Generative AI Model Efficiency in Chief Information Security Officer (CISO) Tasks
摘要
This article presents a comparative analysis of generative AI (GenAI) model efficiency in supporting Chief Information Security Officer (CISO) tasks across operational (Op), technical (Tech), human (Hum), and physical (Phy) security domains. The study evaluates multiple GenAI models (GPT-4, GPT-4o, O1, O3-mini, O3-mini-high) by assessing their performance in handling security-specific prompts related to data analysis, anomaly detection, and regulatory compliance. A structured methodology was developed using Lithuanian and EU cybersecurity legislation to define evaluation criteria and domain-specific requirements. Models were tested through repeated trials using uniform prompts, with expert evaluation focused on response accuracy, consistency, and contextual relevance. Token usage was analysed to assess computational efficiency, revealing that higher token counts do not always correlate with better performance. Findings indicate significant variability in model responses due to stochastic behaviour and differences in architectural design, highlighting the need for human oversight in security-critical applications. The study concludes that while GenAI holds promise as a CISO support tool, it must be integrated cautiously and with clearly defined boundaries.