Intrusion Detection Systems (IDS) based on machine learning (ML)/deep learning (DL) have been developed to detect malicious activities within communication networks. However, existing ML/DL-based IDS are vulnerable to adversarial traffic, which is generated by deliberately adding perturbations to normal traffic in order to maximize classification error. Most of the existing adversarial traffic generation methods overlook the correlations between data features. Failing to account for these correlations results in unrealistic adversarial samples that deviate from the original data distribution. In this paper, we propose Constraint-based Adversarial traffic Generation (CAG), a novel scheme that is capable of evading ML/DL-based IDS while preserving data correlations. Compared to the existing mechanisms, CAG successfully takes the linear and zero multiplication correlations involving three features into consideration. Our experimental results indicate that CAG can evade ML/DL-based IDS in both white-box and black-box attack scenarios.