Pretrained language models (PLMs) are widely used by dialogue systems to generate high-quality responses. Adversarial samples can cause pretrained dialogue models to generate unexpected responses that influence the deployment of the system since malevolent responses can cause the breakdown of the dialogue. Exploring adversarial attacks is important to understand hidden risks and improve victim models. We propose the dialogue malevolence attack task and formulate it as a constrained Markov decision process. We also propose a two-stage reinforcement learning framework to find vulnerabilities of dialogue systems based on pretrained language models (PLMs). Experiments show that the proposed attack framework can effectively attack pretrained dialogue models with high-quality adversarial samples. Warning: this paper contains examples that may be offensive or upsetting.