Comparing GenAI platforms on cybersecurity management task performances

What is it about?

The persistent shortage of cybersecurity professionals, coupled with the consistent increase and complexity of cyberattacks, requires a novel examination of the processes and tasks performed by those professionals in order to cope with the workloads. Research shows that Artificial Intelligence (AI) tools often target technical rather than managerial tasks, highlighting the need for continued human involvement in cybersecurity management. This experimental research study examined how using Generative AI (GenAI) for cybersecurity managerial tasks has the potential to assist in reducing human errors and perform repetitive tasks, thus reducing the cybersecurity managerial loads, allowing them to focus on more strategic aspects of their work. This experimental research study used five GenAI platforms: ChatGPT, CoPilot, Gemini, MetaAI, and Claude. Each GenAI platform generated a real-life scenario and guidelines for cybersecurity managers associated with a managerial cybersecurity task and then cross-evaluated the scenarios and guidelines following predetermined metric measures of (1) Relevancy, (2) Accuracy and reliability, (3) Completeness, as well as (4) Clarity. Scores were generated by each of the five GenAI platforms based on the four metric measures and ranged from 1 = very low to 10 = very high. Then the scores were averaged across all measures and all five GenAI platforms for an overall metrics score ranging from 1 to 10. Analysis of Varicance (ANOVA) was conducted to test for mean differences. Our experimental results indicated that there was a statistically significant mean differences in the scores received between all scenarios (F=7.841, df=4, p<0.001). Specifically, the scenario generated by Claude achieved the highest overall average score (9.3), followed by Gemini (9.0), MetaAI (8.9), ChatGPT (8.7), and CoPilot (8.5). In general, the scenario generated by Claude consistently performed well across all metrics by all five GenAI platforms. The rapid integration of GenAI capabilities into everyday activity may suggest that cybersecurity managers must currently be trained to use AI tools in their daily operations to alleviate their workloads. Having said that, some ethical issues and risks of using GenAI for cybersecurity managerial tasks must be further studied. The cybersecurity workforce shortage was reported to exceed four million workers worldwide in 2024 and is estimated to exceed five million by the end of 2025. Thus, it is significant to further understand the role of AI in improving the efficiency of managerial cybersecurity tasks. The value of this research lies in three facets. First, the demonstration of using GenAI to perform managerial cybersecurity tasks. Second, the novel methodology, in which the GenAI platforms assess the outputs by self- and cross-evaluating them. Finally, our development of novel metrics to assess managerial tasks can be of great value for researchers and industry.

Featured Image