As IT advances, cybersecurity is increasingly critical, especially in China and the U.S. While LLMs have excelled in many domains, their complexity and dynamism pose significant challenges in cybersecurity. Current benchmarks face issues like limited task types, poor language support, unclear difficulty levels, and lack of prompt quality evaluation, hindering effective assessment. To address these limitations, we propose SecBen, a more comprehensive cybersecurity benchmark and open leaderboard. It is the first dynamic, scalable cybersecurity benchmark supporting any data labels and associated metrics. Specifically, SecBen supports both Chinese and English, based on Bloom’s taxonomy, covering three skill levels (CyberKUT, CyberNLP, CyberDSA), with a focus on the tasks like Q&A, NER, and code understanding, encompassing nearly all NLP types from 12 datasets totaling 18K instances. We extensively evaluate 14 popular and advanced LLMs on SecBen, including 4 bilingual general LLMs, 8 Chinese-focused LLMs, and 2 cybersecurity domain LLM. The results reveal notable performance differences: the large-parameter, closed-source ERNIE-3.5-8K excels, while cybersecurity data fine-tuned LLMs like AutoAudit and SecGPT show only limited improvements, nearly all LLMs struggle with text reasoning tasks ( https://github.com/LthreeC/SecBen ).