Gastrointestinal stromal tumors (GIST) are a type of prevalent tumor with potential malignant properties. Given the prominent characteristic that the morphological characteristics (such as size and edge) and anatomical location of gastrointestinal tumors are strongly correlated with tumor types, existing technologies often suffer from limited accuracy in classification and segmentation due to incomplete feature extraction or interference from redundant information. Therefore, this paper proposes a multi-task voting feature transformation network (VFT-Net). This network integrates segmentation and classification tasks through a multi-task learning framework, and designs a multi-scale feature extraction module, Task Voting Gate (TVG), and Feature Transformation Pyramid (FTP) to achieve efficient feature interaction. Experimental results show that the Dice coefficient of VFT-Net on the GISTS dataset reaches 0.923 ± 0.042, which is an improvement of more than 3.9% compared with existing algorithms. The detail segmentation accuracy (Precision, Recall) is increased by 8.6% and 3.1% respectively, and the classification accuracy reaches 0.97, verifying its effectiveness and advancement in the tasks of gastrointestinal tumor segmentation and classification.