Classification and segmentation of breast and thyroid tumors are crucial for accurate diagnosis and treatment planning in medical applications. However, current approaches often treat these tasks separately, overlooking the potential of joint learning to enhance model performance. To address this gap, we propose InstCNet, a dual-branch network designed for joint learning of instance segmentation and classification tasks for tumor diagnosis. At the core of InstCNet is the PVSSBlock, which integrates global semantic modeling and progressively fuses multi-scale features within the instance segmentation network. Additionally, knowledge of edge contours and surrounding tissues is incorporated to enhance the network’s performance. Specifically, tumor contours and texture features learned by the classification branch are transferred to the instance segmentation branch, preserving critical tumor edge information that may otherwise be lost during feature extraction. Furthermore, we designed a multi-scale cross attention (MSCA) mechanism to unify the information streams from classification and segmentation tasks to align tumor semantic information across multiple scales. Experimental results demonstrate that the proposed InstCNet outperforms state-of-the-art models in both tumor classification and segmentation tasks.