Effective communication between doctors and patients is pivotal for accurate diagnosis and its corresponding treatment planning. However, language barriers, misinterpretations, and inexpressibility pose a challenge for doctors, particularly in multilingual environments like rural India. To address this challenge, we demonstrate a Natural Language Processing-based framework for extracting clinically relevant information at the token/word level from doctor-patient conversations, which forms a firm foundation for accurate medical information extraction frameworks which can be customized as per requirement. We assess pre-trained large language models on a custom annotated dataset by evaluating them using word-level performance metrics. Our results demonstrate better performance of domain-specific models like BioBERT and ClinicalBERT for entities with greater biomedical importance, such as medical conditions and symptoms, and that of BERT, RoBERTa, and DistilBERT for general entities such as a person’s age and time frame, etc. Models are tested on real-life examples gathered, and the adaptability is assessed. Our work introduces one of the few comprehensive NER frameworks specifically tailored for information extraction from spoken or informal medical dialogue in low-resource settings, aiming to improve healthcare accessibility at scale. This work underscores the potential of NER in improving medical communication and supports the development of intelligent systems for the precise extraction of information in diverse healthcare settings.