Rapid and accurate diagnosis of repository diseases like COVID-19 and viral pneumonia using chest X-rays (CXRs) is vital. Traditional diagnostic methods often face challenges due to variability in image quality and subtle disease manifestations. This study introduces a hybrid multi-stage network for lung segmentation, disease classification, and severity localization from CXR images. Using the COVID-19 Radiography Database, which includes 3616 COVID-19, 10,192 normal, and 1345 viral pneumonia images, the network employs a U-Net for lung segmentation, followed by disease classification using ResNet and localization with Grad-CAM. Experiments results highlight the network’s robust performance across multiple architectures, demonstrating high accuracy in segmentation and classification tasks. The proposed method enhances diagnostic precision and offers clear visual insights into disease severity supporting effective and timely medical intervention.