Once deployed in production, Natural Language Processing (NLP) models may receive adversarial inputs that lead to erroneous or unintended predictions. These inputs contain textual perturbations that may appear benign to humans but cause models to fail. An effective defense for improving robustness against such textual data without requiring model retraining is adversarial purification. The approach transforms input text to mitigate the influence of adversarial perturbations before they reach the end prediction models. To this end, this paper introduces a novel adversarial purification method for textual data using Large Language Models (LLMs). Specifically, the proposed method leverages decoder-based LLMs and their in-context learning capabilities to guide the transformation of input text. First, a set of adversarially perturbed examples is automatically generated from a collection of benign texts. These benign-adversarial pairs are then used as few-shot demonstrations within an LLM instruction prompt to guide the transformation of new input texts and reduce the impact of adversarial perturbations. Experiments were conducted to assess the effectiveness of the proposed method and demonstrated an advantage over several well-known LLM-based defense approaches. Specifically, the proposed method demonstrates robustness improvement by up to 10% relative to the best zero-shot purification defenses. It corrects many previously misclassified adversarial examples while preserving benign accuracy. Overall, the proposed approach paves the way for safer NLP systems in real-world deployments.