Accurate diagnosis of pulmonary diseases from chest X-rays remains a challenging task, especially due to the scarcity of labeled data. In this work, we propose an enhanced zero-shot learning framework that integrates BioBERT—a pre-trained model for biomedical text representation—into a contrastive learning pipeline for medical imaging diagnostics. Our method leverages the synergy between textual radiology reports and image data by combining BioBERT with a CLIP-based architecture. To address the lack of textual data in standard datasets, we generate synthetic radiology reports for each pathology, aiming to mimic the complexity of actual clinical descriptions. Through extensive experiments, we demonstrate significant improvements in disease classification accuracy and generalization, particularly in zero-shot inference scenarios. Our analysis also reveals how similarities in pathology descriptions can lead to misclassifications, emphasizing the importance of nuanced textual representations. This work establishes a zero-shot learning method in medical imaging, highlighting the potential of BioBERT in enhancing automated diagnostic systems for chest X-rays.