Accurate medical image segmentation is challenging due to the high variance of out-of-distribution (OOD) data, which is costly for acquisition and annotation. However, existing methods often focus on the sub-scenarios of the OOD problem (e.g., Domain generalization) and approach this via a pretraining and fine-tuning domain adaptation paradigm, which does not explicitly utilize the intrinsic semantic relationship among those OOD tasks. To address this problem, we introduce a novel Attribute-Based Segmentation (CA-Seg) method to unify the OOD problems, where only the OOD object class semantic information is required to bridge the domain gap. CA-Seg contains two stages of learning: high-level semantic abstraction learning and low-level visual pattern learning. In the first stage of the training phase, we extract morphological knowledge of the tissue/organ of interest using an off-the-shelf Vision-Language Model (VLM), leveraging its rich language and image pattern association ability to describe class semantics into a set of human-understandable text attributes. Subsequently, the concepts serve as a condition to guide the low-level visual pattern learning by using the flow-matching or dice loss. During the adaptation phase, CA-Seg only requires the OOD class labels to rebind the attributes and the target object class to achieve good segmentation performance. CA-Seg addresses the limitations of data scarcity in the broad OOD problems and is computationally efficient in adapting to the OOD data, making it ideal in resource-constrained settings. We evaluated CA-Seg on 3 common OOD tasks in medical image segmentation, demonstrating a high cross-domain segmentation performance with limited data availability. Our code is available at https://github.com/iClaude1998/CA-seg .