Deep learning models achieve strong performance in vascular segmentation but often lack transparency, a critical factor for clinical adoption. We introduce a novel explainability framework combining graph-based point selection and blob-level analysis of saliency maps to interpret model behavior. By linking anatomically relevant points from the vascular graph to localized attributions, we evaluate how model predictions align with domain-specific features such as tubularity, connectivity, and thickness. Our analysis on two vascular datasets reveals that model decisions are primarily driven by local visual cues, with limited evidence of global anatomical reasoning. These findings highlight the need for structured explainability tools and architectures that better integrate anatomical context in medical image segmentation.