Surgical scene segmentation is an important task in the automated analysis of surgical videos and medical images. Traditional techniques for this task suffer from severe data scarcity characteristic of the medical domain. To address this challenge, contemporary research utilizes pre-trained models, which are finetuned using available data. Nonetheless, this approach involves repeated training of models with millions of parameters whenever new data becomes available. A recent foundational model called Segment Anything (SAM) has demonstrated impressive generalization capability in natural images, offering a potential resolution to this challenge. However, its applicability to the medical domain is hindered by its compute-intensive training along with need for task-specific prompts. These prompts can be bounding-boxes or foreground/background points, mandating expert annotation of every image. Expert annotation is not feasible as data volume increases. In this study, we propose LoRASAM - a highly efficient adaptation of SAM that enables text-guided segmentation of medical images. LoRASAM uses the labelname for precise image segmentation. To facilitate finetuning of LoRASAM, we utilize low-rank adaptation, which reduces the number of training parameters by more than 99% as compared to SAM, while significantly improving performance (about 70%). Extensive experimentation over three public surgical-scene datasets validates the superiority of LoRASAM over existing state-of-the-art methodologies. In addition, we show similar gains for non-surgical modalities such as x-ray and ultrasound. Our approach has the potential to enhance the segmentation performance and curtail expert involvement while adapting SAM for niche applications. Code: https://github.com/JayParanjape/LoRASAM