PURIFIER, a representative defense against label-only membership inference attacks, perturbs confidence scores to protect training data privacy but exhibits critical limitations: its KNN-based label swapper suffers from degraded accuracy on small training sets; its confidence reformer statically emulates a fixed non-member reference distribution, undermining the defense effectiveness when real-world non-member data drifts; and its variational autoencoder introduces excessive distortion that unnecessarily degrades model utility. To address these, this paper proposes ACM. To address the first limitation, an adaptive label swapper, leveraging an autoencoder for efficient and stable membership discrimination, selectively swaps the predicted labels to counter label-only attacks. To address the second and third limitations, we design an adversarial confidence modifier based on a CAE. First, it maps member confidence scores into the non-member space. Finally, adversarial training aligns the distributions by optimizing a combined loss function, ensuring the modified member scores are statistically indistinguishable from those of non-members. Extensive experiments show ACM outperforms PURIFIER, improving membership discrimination accuracy by 40.3% and reducing utility loss by 18%–21%. ROC analysis confirms ACM reduces the attack AUC to 0.49, making membership inference statistically equivalent to random guessing.