Underwater image semantic segmentation is widely used in the recognition and navigation of vision-guided underwater robots. However, due to issues such as insufficient underwater scene illumination and turbidity of the water, the contrast between targets and backgrounds is low. Existing underwater semantic segmentation methods ignore the recognition differences between underwater images, making it challenging for models to extract sufficiently robust visual features from noisy underwater images. To address this problem, we propose SEA-Net, which incorporates a SEA-Adapter and Visual Prompt Tuning. In the framework, we use a severity metric to address various complex noise problems in underwater images. The severity metric classifies all underwater images into High and Low-severity images and combines the Visual Prompt Tuning of two-branch alternating training, this allows the model to learn more robust visual features from different perspectives. On both the SUIM and DeepFish benchmarks, our proposed SEA-Net outperforms state-of-the-art methods in underwater image semantic segmentation tasks.