Accurate and efficient anomaly detection in medical imaging plays a crucial role in advancing computer-aided diagnosis. This study introduces a multimodal detection approach that utilizes both image and text features to enhance anomaly recognition in medical images. We formulate detection problem across various medical imaging modalities and address the detection task by using both multimodal information within detection framework. The proposed framework utilizes image and text features through cosine similarity-based learning to enhance object’s semantic understanding. The text features are generated from bounding box annotations, focusing on attributes such as category, location, size, and area, which are derived from existing annotations, eliminating the need for additional labeling efforts. During inference, we adopt a text-free approach, eliminating the reliance on text annotations at deployment, unlike many existing frameworks that require text-based guidance. This design ensures no increase in annotation burden or parameters during inference. Experimental results demonstrate significant performance improvements across multiple datasets, highlighting the efficiency and effectiveness of the proposed framework in enhancing anomaly detection without additional computational or annotation costs.