The training of general image segmentation models faces two major challenges. First, heterogeneity among different modalities and tasks requires the model to handle data from diverse sources. Second, each medical dataset is typically single-modal and only partially labeled, so the model needs to integrate datasets with partial labels to gain a general capability for different modalities and different subjects. In this paper, we present a novel model architecture that addresses the challenges of multimodal and multitask medical image segmentation by decomposing tasks into modality-specific and task-specific components. Our approach leverages specialized expert pathways to extract features from each modality and dynamically balances feature extraction through modality labels and task descriptors. Modality-related feature extraction is embedded in the encoding procedure, and task-specific segmentation is achieved by feature fusion during decoding. Transformer based attention mechanism is utilized to process an arbitrary number of regions of interest (ROIs), offering greater flexibility than conventional models. The integration of multimodality and multitask fusion modules enhances both interpretability and generalization for medical image segmentation. We validate our approach on three datasets, including FLARE22, AMOS22, and ATLAS23, demonstrating superior performance compared to both single-task and universal multitask models.