Network intrusion detection systems (NIDS) are increasingly developed using machine learning (ML) techniques. However, incorporating ML into NIDS introduces a new vulnerability: the threats and limitations arising from adversarial machine learning (AML) attacks. Specific to this application domain, AML could enable an attacker to disguise incoming malicious packets and fool a NIDS to classify them as benign. Although AML has been researched actively in other domains, assessing its impact in networks remains an outstanding challenge, especially since network protocols pose a constrained domain for adversarial packet generation. More specifically, there is a need to experiment with the latest advances in AML attacks – usually developed for an unconstrained domain – on such domains. This paper presents a novel approach to this problem, where a variety of attacks could still be applied and correctly evaluated in a constrained domain. We show an implementation of this approach for NIDS, by developing an adversarial packet validator for different network protocols. By conducting extensive experiments using multiple data sets, ML models, and attacks, we show how our approach can bridge the gap between progress in AML and a constrained domain like NIDS. Evaluation enabled by our approach and its implementation suggests that black-box evasion attacks continue to be a threat to NIDS, despite many constraints offered by this domain.