Ultra-dense network (UDN) deployment in 5G or B5G systems creates significant handover management challenges due to frequent cell transitions and highly dynamic signal conditions. While existing deep reinforcement learning approaches optimize handover decisions based on signal quality, they treat all scenarios uniformly and focus on single objectives. This paper proposes a KPI-Adaptive Deep Q-Network (DQN) framework that addresses these limitations through three key contributions. First, a multi-objective reward function with adaptive penalty weights is incorporated that follows multi-objective reinforcement learning principles and enable explicit balance between Radio Link Failure (RLF), handover frequency, and signal quality. Second, inside DQN, a phase-based curriculum learning mechanism with three distinct training stages is introduced that progressively shifts priorities from connection stability to balanced optimization. Third, a dynamic weight adaptation mechanism is implemented that automatically adjusts objective weights based on real-time KPI performance deviations from target values. Comprehensive simulation results in a 21-cell ultra-dense environment demonstrate that the proposed method achieves 13.2% RLF reduction compared to baseline 3GPP A3 approach and 5.4% improvement over plain DQN, while simultaneously reducing handover frequency by 1.8% and maintaining stable signal quality above 70 dB throughout 120 training episodes.