During the past three decades, numerous studies have proposed various network intrusion and anomaly detection methods and techniques. The majority of these research studies are only limited to the performance evaluation of intrusion detection techniques by using benchmark datasets. The performance of any intrusion detection system mainly relies on the quality of IDS dataset used for discriminating normal traffic from attack traffic. In the present scenario, one of the most challenging issues for an intrusion detection system is to precisely detect modern day DDoS attacks. Existing IDSs for detecting network intrusions are not suitable for DDoS attack detection as the underlying datasets used for detection do not satisfy the evaluation criteria of a quality IDS dataset. In this research, we have considered the eleven benchmark IDS datasets between 1998 and 2022 widely used in various research studies and evaluated them by applying Gharib’s IDS evaluation framework to gauge the quality of IDS datasets. We show in detail how various benchmark datasets can be evaluated with detailed computations carried for each benchmark dataset. The present study gains importance as it considers the attack diversity information recently published in the NetScout 1H 2023 DDoS threat report. The evaluation study proved that among all the eleven datasets, CICDDoS2019 dataset satisfied all eleven criteria defined in the evaluation framework for a quality IDS dataset to detect modern DDoS network attacks by learning algorithms.