Continuous external disturbances impose a significant influence on humanoid robot locomotion, causing command execution failures. Limited by the absence of force information, these controllers are limited to passive recovery rather than active adjustment. To address this issue, this study proposes RL-Force, an asymmetric reinforcement learning framework that integrates curriculum learning with a co-trained external force estimator using only proprioceptive sensors. The estimator infers a force vector and a point of application, thereby assisting the policy in adapting to external forces. Simulated experiments with a self-built humanoid robot demonstrate superior velocity tracking and adaptive posture adjustment compared to baseline methods. The saliency analysis underscores the critical role of force information in policy inference, improving recovery from disturbances.