Accurate pneumonia segmentation is crucial for effective diagnosis and treatment of pneumonia, a major global health concern. While vision-based medical image segmentation models have shown promise, their heavy reliance on expert-level pixel annotations limits clinical use. Multi-modal segmentation methods aim to mitigate annotation needs by incorporating text guidance, yet face persistent challenges in optimally aligning high-resolution visual features with abstract textual features due to their inherent representational gap. To alleviate these issues, we introduce VLSCAM, a Vision-Language Sliding Cross Attention Model, which utilizes a newly proposed Sliding Cross Attention mechanism to achieve more rational utilization of both image and text features and more accurate pneumonia segmentation. Extensive experiments on the QaTa-COV19 and MosMedData+ datasets demonstrate that VLSCAM outperforms both uni-modal and multi-modal methods. Furthermore, even with only 5% of the training data, our method still outperforms the best uni-modal method. Further experiments have demonstrated the effectiveness of our proposed method and its potential in reducing annotation costs for medical segmentation tasks. Code will be available.