Audio-visual Generalized Zero-shot Learning (AV-GZSL) presents a complex task, where the goal is to classify video content from both seen and unseen categories using multi-modal audio-visual data. Recent research efforts have mostly incorporated cross-modal attention of modalities and textual label embeddings to tackle this complex task. However, existing approaches often prioritize performance on unseen classes, ignoring the significance of knowledge understanding of seen classes. To address this imbalance, we introduce the Negative Adjustment Contrastive Learning (NACL) scheme. During each training epoch, NACL precisely selects negative samples based on the performance of the last epoch. This approach helps the model mitigate bias between seen classes used for training, resulting in a more comprehensive knowledge transfer to unseen classes. Moreover, many research efforts commonly adopt a benchmark with a two-stage training and evaluation protocol. Yet we find that this protocol lacks optimality in hyperparameter selection, potentially resulting in sub-optimal performance. Therefore, we propose a new dataset splitting method, replacing the two-stage approach with a more effective one-stage training protocol. By employing the NACL scheme and implementing our revised protocol, we observe a substantial improvement in performance compared to existing SOTA works on three widely used audio-visual datasets.