The robot soccer game is a classic problem in robot control, which can design and verify new algorithms and migrate to other typical robotic applications. Most commonly applied methods are still rule-based, and their performance may be limited as the number of players increases and the interaction between players becomes more and more intensive. Reinforcement-learning-based methods are gaining attention and have been applied to deal with high-dimensional problems with various constraints, like robot soccer. This paper proposes a RL-based method to train a humanoid robot to learn and kick a soccer ball into a gate from scratch. To enhance performance, this paper designs a novel form of a reward function and generalizes the case to obstacle presence scenes. Simulation studies are performed based on Isaac Gym, and the result shows substantial gains in goal accuracy, efficiency, and speed compared with the baseline.