Class-Incremental Learning (CIL) is a setting of continual learning in which models must continuously learn new tasks while retaining previously acquired knowledge, mitigating catastrophic forgetting. In this work, we introduce a framework, FPFA, which leverages the robust vision-language encoder of CLIP for classification. In continual learning scenarios, new classes may resemble previously learned ones, increasing the risk of misclassification. To address this, we propose a feature alignment mechanism to ensure that the representation of easily confused classes is pushed apart while aligning the image and text embeddings of previously learned classes more closely. For further enhancing knowledge retention, we incorporate Fisher-weighted parameter fusion, which is applied after each task. This combination effectively realigns textual and visual embeddings to reduce interference among semantically similar classes while preserving critical parameters to maintain prior knowledge. Experimental results on CIFAR100 and TinyImageNet datasets indicate that our framework significantly outperforms current state-of-the-art techniques in CIL by reducing forgetting and enhancing adaptation to new tasks. The proposed approach highlights the potential of combining adaptive representation adjustment with Fisher-weighted fusion to advance continual learning performance in vision-language models.