Anomaly segmentation aims to identify anomalous regions in images that do not belong to any known category, while road anomaly segmentation specifically detects and precisely segments abnormal regions from road scene images. Existing methods typically rely on fine-tuning with negative samples and threshold strategies for road anomaly segmentation. However, in real-world settings, these approaches face two major challenges: (1) the long-tailed distribution of anomalous data limits domain generalization. and (2) the threshold method often fails to balance detection accuracy with noise suppression. To address these issues, we propose a novel Dual-Branch Anomaly Segmentation framework that combines panoramic segmentation with open-set object detection. The framework effectively improves the accuracy and robustness of road anomaly segmentation by combining Prompt-Aligned Detection (PAD) branch and Known-Class Filtering (KCF) branch. Specifically, the PAD branch adopts text-guided object detection to generate the bounding box of anomalous objects, which guides the segmentation model to realize anomaly segmentation. The KCF branch removes the responses of known classes through the class filtering mechanism, thus realizing the pixel-level segmentation of unknown objects. By fusing the outputs of the two branches, our method attains fine-grained segmentation of anomalous regions. The DBAS framework achieves state-of-the-art performance on multiple benchmarks, including Road Anomaly, Fishyscapes Static, SMIYC-Anomaly, and SMIYC-Obstacle datasets. The code is available at https://github.com/fugit0316/DBAS .