Before deploying autonomous vehicles (AV), comprehensive safety assessments are essential. To enhance efficiency, generating safety-critical scenarios to accelerate testing is crucial. This paper proposes an adversarial strategy based on deep reinforcement learning to generate safety-critical scenarios for evaluating autonomous vehicle performance. First, a highway ramp merging scenario is modeled using a Markov process. Second, deep reinforcement learning is used to train the background vehicle (BV) to generate adversarial behaviors against the autonomous vehicle. A reasonable reward mechanism is introduced to prevent extreme dangerous behaviors and ensure scenario rationality. Simulation results show that the generated safety-critical scenarios significantly increase collision rates and reduce the performance of AV. Additionally, this method can generate scenarios with varying levels of danger, supporting more refined testing.