EVALUATING THE OFFENSIVE UTILITY OF LARGE LANGUAGE MODELS IN CYBER THREAT GENERATION
- Authors
-
-
Mukhammadbek Bakhtiyorov
Student of Academic Lyceum under Bukhara State Technical University -
Jamolbek Mattiev
Computer Science and AI Technologies department, Urgench State University named after Abu Rayhan Biruni
-
- Keywords:
- Large Language Models cyber threat generation offensive cybersecurity Cyber Threat Intelligence threat modeling attack-path reasoning retrieval-augmented generation reliability operational utility cybersecurity automation
- Abstract
- Large Language Models (LLMs) are increasingly integrated into cybersecurity workflows, creating both defensive opportunities and new forms of offensive enablement. Existing research demonstrates capabilities in threat modeling, attack-path construction, Cyber Threat Intelligence (CTI) processing, adversarial text generation, and automated penetration testing; however, these capabilities are commonly evaluated in isolation. This review synthesizes recent evidence and develops a methodological framework for evaluating offensive utility as a multidimensional construct encompassing technical capability, correctness, contextual grounding, reliability, operational relevance, temporal efficiency, and human-expertise compression. The review distinguishes linguistic plausibility from cyber validity and separates model capability from system-level utility. Evidence from recent studies indicates that LLMs can support attack-technique mapping and multi-stage reasoning, while retrieval-augmented architectures can improve contextualization. At the same time, hallucination, inconsistent reasoning, latency, safety restrictions, and retrieval dependence constrain practical utility. The proposed framework therefore treats offensive utility as an emergent property of the interaction among model, context, retrieval, tools, and human oversight rather than as a single model attribute. The framework provides a basis for reproducible comparative evaluation without requiring unauthorized real-world offensive activity.
- References
-
[1] M. S. Salek et al., ‘A large language model-supported threat modeling framework for transportation cyber-physical systems’, IEEE Access, vol. 13, pp. 163046–163070, 2025, doi: 10.1109/ACCESS.2025.3603580.
[2] Y. R. Chen, M. J. Cui, D. Wang, and others, ‘A survey of large language models for cyber threat detection’, Computers & Security, vol. 145, p. 104016, 2024, doi: 10.1016/j.cose.2024.104016.
[3] A. Tellache, A. A. Korba, A. Mokhtari, H. Moldovan, and Y. Ghamri-Doudane, ‘Advancing autonomous incident response: Leveraging LLMs and cyber threat intelligence’. 2025. [Online]. Available: https://arxiv.org/abs/2508.10677
[4] Y. Zhang et al., ‘AttacKG+: Boosting attack graph construction with Large Language Models’, Computers & Security, vol. 150, p. 104220, 2025, doi: 10.1016/j.cose.2024.104220.
[5] J. Jiang, S. K. Jha, and others, ‘Can LLM-generated misinformation be detected: A study on Cyber Threat Intelligence’, Future Generation Computer Systems, vol. 173, p. 107877, 2025, doi: 10.1016/j.future.2025.107877.
[6] Z. Liu, J. Shi, and J. F. Buford, ‘CyberBench: A multi-task benchmark for evaluating large language models in cybersecurity’, in AAAI-24 Workshop on Artificial Intelligence for Cyber Security (AICS), 2024. [Online]. Available: https://arxiv.org/abs/2403.18493
[7] M. Bhatt et al., ‘CyberSecEval 2: A wide-ranging cybersecurity evaluation suite for large language models’. 2024. [Online]. Available: https://arxiv.org/abs/2404.13161
[8] S. Wan et al., ‘CyberSecEval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models’. 2024. [Online]. Available: https://arxiv.org/abs/2408.01605
[9] S. Shafee, A. Bessani, and P. M. Ferreira, ‘Evaluation of LLM-based chatbots for OSINT-based cyber threat awareness’, Expert Systems with Applications, vol. 261, p. 125509, 2025, doi: 10.1016/j.eswa.2024.125509.
[10] P. R. B. Houssel, S. Layeghy, P. Singh, and M. Portmann, ‘eX-NIDS: A framework for explainable network intrusion detection leveraging large language models’, Computers and Electrical Engineering, vol. 129, p. 110826, 2026, doi: 10.1016/j.compeleceng.2025.110826.
[11] S. Shafee, A. Bessani, and P. M. Ferreira, ‘False alarms, real damage: Adversarial attacks using LLM-based models on text-based Cyber Threat Intelligence systems’, Future Generation Computer Systems, vol. 184, p. 108603, 2026, doi: 10.1016/j.future.2026.108603.
[12] P. Balasubramanian et al., ‘Generative AI for cyber threat intelligence: Applications, challenges, and analysis of real-world case studies’, Artificial Intelligence Review, vol. 58, p. 336, 2025, doi: 10.1007/s10462-025-11338-z.
[13] M. A. Ferrag et al., ‘Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities’, Internet of Things and Cyber-Physical Systems, vol. 5, pp. 1–46, 2025, doi: 10.1016/j.iotcps.2025.01.001.
[14] M. Uddin et al., ‘Generative AI revolution in cybersecurity: A comprehensive review of threat intelligence and operations’, Artificial Intelligence Review, vol. 58, p. 236, 2025, doi: 10.1007/s10462-025-11219-5.
[15] H. Alqahtani and G. Kumar, ‘Large language models for cybersecurity intelligence: A systematic review of emerging threats, defensive capabilities, and security evaluation frameworks’, Computers, Materials & Continua, vol. 87, no. 3, 2026, doi: 10.32604/cmc.2026.077367.
[16] F. Tang, H. Li, M. Zhao, Z. Wu, S. Peng, and T. Yin, ‘LRCTI: A large language model-based framework for multi-step evidence retrieval and reasoning in cyber threat intelligence credibility verification’. 2025. [Online]. Available: https://arxiv.org/abs/2507.11310
[17] A. Happe and J. Cito, ‘On the surprising efficacy of LLMs for penetration-testing’. 2025. [Online]. Available: https://arxiv.org/abs/2507.00829
[18] G. Deng et al., ‘PentestGPT: An LLM-empowered automatic penetration testing tool’. 2023. [Online]. Available: https://arxiv.org/abs/2308.06782
[19] M. Bhatt et al., ‘Purple Llama CyberSecEval: A benchmark for evaluating the cybersecurity risks of large language models’. 2023. [Online]. Available: https://arxiv.org/abs/2312.04724
[20] D. Bhusal et al., ‘SECURE: Benchmarking large language models for cybersecurity’, in Proceedings of the 2024 Annual Computer Security Applications Conference, 2024. doi: 10.1109/ACSAC63791.2024.00019.
[21] Á. Ruiz-Ródenas, J. Pujante Sáez, D. García-Algora, M. Rodríguez Béjar, J. Blasco, and J. Hernández-Ramos, ‘SynthCTI: LLM-driven synthetic CTI generation to enhance MITRE technique mapping’. 2025. [Online]. Available: https://arxiv.org/abs/2507.16852
[22] J. Zhang et al., ‘When LLMs meet cybersecurity: A systematic literature review’, Cybersecurity, vol. 8, p. 55, 2025, doi: 10.1186/s42400-025-00361-w.