Medical images, particularly chest X-rays, are important information for diagnosing pulmonary diseases such as pneumonia, tuberculosis, and lung cancer. However, the inherent limitations of X-ray images, such as low resolution and the inability to integrate with patient medical history, affect diagnostic accuracy, especially in cases of early-stage or subtle lesions. In this study, we aim to bridge the gap between textual symptom descriptions and visual X-ray representations by leveraging advanced generative modeling techniques. Specifically, we propose SLG-Flow, an innovative framework that combines text-based latent encoding with cross-modal flow matching techniques to generate high-quality, clinically relevant chest X-ray images. SLG-Flow is built upon the fundamental components of a Variational Autoencoder (VAE) and incorporates a flow matching mechanism to directly map textual latent representations to image latent space. By adopting a text-aware tokenizer (TA-TiTok) and fine-tuning using Low-Rank Adaptation (LoRA), this framework can efficiently adapt to the medical imaging domain while retaining general domain knowledge. Quantitative evaluations demonstrate that SLG-Flow excels in terms of image quality and diagnostic relevance (FID: 19.00, CLIPscore: 34.65).