Automated Short Answer Grading (ASAG) comes under automatic answer script evaluation where the answer length is limited from one phrase to one paragraph. The main task in ASAG is generating a good sentence embedding for both the student and the reference answers. The existing works on the embedding creation perform better when using different deep-learning techniques and language models. However, the deep-learning techniques’ performance mainly depends on the training set size and quality. Most of the publicly available datasets typically have a limited number of reference and student answer pairs. To automate the dataset expansion, text augmentation techniques can be used. Conventional methods like back-translation, synonym replacement, and random deletion may replace some important technical words with other non-relevant terms, resulting in a loss of contextual meaning. We propose a new augmentation strategy for the ASAG datasets using LLM (Large Language Model) prompting. The effect of the proposed strategy is analysed on sentence transformer fine-tuning. We experimented with four different sizes of augmented training sets to determine the impact of the size of augmented training data on fine-tuning the sentence transformer model. Results indicate that sentence transformer fine-tuned using a 50% prompt-driven augmented dataset generates better embeddings. After having good embeddings, the traditional classifiers can be used to classify the student answers to different scores. We introduce “Prompt Adaptive Oversampling (PAO)” to address the class imbalance issue during grade classification. The effectiveness of the proposed strategy is analysed on two different public datasets: SPRAG, and Mohler-ASAG. The proposed method performs better while training highly imbalanced datasets. The source code of this work is available here.