Medical hyperspectral imaging (MHSI) is a highly promising technology, offering new opportunities for computational pathology and precision medicine. However, the high spatial-spectral dimensions make simultaneous consideration of spatial and spectral features for image segmentation extremely challenging. In this study, we propose a segmentation network that slices the high-dimensional features of hyperspectral images into low-dimensional sequences, thus analyzing the spectral characteristics of medical hyperspectral images from a sequence perspective. This network is capable of learning both spectral and spatial information simultaneously, thereby enhancing the performance of image segmentation tasks. Its uniqueness lies in leveraging the Convolutional Long Short-Term Memory (ConvLSTM) structure to convert the long-range spectral dependencies of MHSI into relationships between low-channel image sequences, significantly improving the model’s inference speed. Experiments conducted on the publicly available Multi-Dimensional Choledoch dataset demonstrate that compared to the state-of-the-art medical hyperspectral image segmentation algorithm, the Dual-Stream algorithm, our approach improved the DSC metric by 0.24%, increased inference speed by 1.4 times, and was 5–20 times faster than existing 3D networks.