The Kudmali language, an official language of West Bengal is a lesser-known and potentially vulnerable language. It encounters significant hurdles in developing Automatic Speech Recognition (ASR) systems due to its limited digital footprint and scarcity of annotated datasets. This paper explores the use of the multilingual XLS-R (Cross-Lingual Speech Representation) model, a transformer-based pre-trained ASR framework, for detecting and transcribing Kudmali speech. We used grapheme-based Kudmali dataset development, offline and online data augmentations and fine-tuning the XLS-R model. As Kudmali language has no accepted script till now, state wise different scripts are used for teaching in universities. Here we used Latin alphabet for transcription. Kudmali voice data were collected and Text To Speech (TTS) data were prepared and speech perturbations were executed over audio files. All speech data were preprocessed through .wav file creation, normalization and other necessary steps. Both augmentations (online and offline) were implemented before and during fine-tuning with Kudmali dataset, which led to perform much better than the baseline model revealing notable improvements. It reduced 25% Word Error Rate (WER) to 19.8% and 18.2% Character Error Rate (CER) to 12.1%, where further improvements obtained through data augmentation techniques. This result establishes that fine-tuning multilingual models like XLS-R with augmentation is a right choice for the development of Automatic Speech Recognition System for low-resource languages like Kudmali, which in turn helps to empower this language and enhance its digital inclusion. If the limitations are overcome, then this study might be helpful for the development of robust NLP tools for Kudmali and other resource-constraint languages. Our study also evaluates the model’s real time processing ability and error rates in recognizing and generating text for this less familiar language and help in developing Kudmali ASR system.