Neoantigens, defined as tumor-specific peptides triggering immune responses, have emerged as a promising therapeutic target. Recently, proteogenomic analysis employing liquid chromatography-tandem mass spectrometry (LC-MS/MS) has become a standard approach for neoantigen identification. However, the accuracy of this method is limited by inconsistent false discovery rate (FDR) estimation and inherent biases in the target-decoy method. Retention time (RT), a key physicochemical property of peptides, provides an effective metric for evaluating identified neoantigens’ quality. Therefore, we propose PepHiFuse, a deep learning framework incorporating global biochemical semantics and local physical interactions to predict peptide RT. Moreover, it employs an adaptive weighting mechanism to balance these complementary feature domains dynamically. Evaluated on three large public datasets, PepHiFuse outperforms three recently published methods (AutoRT, AlphaPeptDeep, PepMNet). The significant improvement in RT prediction accuracy demonstrates the efficacy of PepHiFuse as a prediction tool. PepHiFuse also enhances neoantigen identification by selectively filtering low-confidence candidates while retaining high-quality targets across multiple FDR estimation methods when applied to clinical proteogenomic datasets. This ability emphasizes the practical utility of the PepHiFuse method, offering a critical step toward reliable immunotherapeutic discovery by incorporating accurate RT prediction into neoantigen validation. The code is available at https://github.com/lyotvincent/PepHiFuse .