In a groundbreaking study, researchers have developed benchmark datasets to evaluate the general knowledge of Large Language Models (LLMs) in cybersecurity. The creation of CyberMetric80, CyberMetric500, CyberMetric2000, and CyberMetric10000 has marked a significant step towards understanding the capabilities and limitations of LLMs in this domain.
These multiple-choice QA datasets comprise questions collected from various sources, including NIST standards, research papers, publicly accessible books, RFCs, and other publications. The results have shown that state-of-the-art LLM models, such as GPT4o, GPT4turbo, Mixtral8x7B, Instruct Falcon180BChat, and GEMINIpro 10, outperformed humans on CyberMetric80, although highly experienced human experts still excelled in complex tasks.
The study highlights the importance of balancing human expertise with AI capabilities in cybersecurity. By making the CyberMetric dataset publicly available, researchers can now compare and improve their own LLM models, ultimately accelerating progress in this field. As research continues to advance, it is essential to strike a balance between leveraging AI capabilities and preserving human expertise, enabling more effective solutions for complex cybersecurity challenges.
↧