<p>To address the challenges in the optimal control of modular robot manipulators (MRMs), including dependence on admissible initial control strategy, limited convergence speed, and difficulty in satisfying persistent excitation conditions, this paper proposes an integral-reinforcement-learning-based novel hybrid iterative algorithm via experience replay. First, a state-space description of the MRM subsystem is established under the joint torque feedback framework, providing a unified dynamic basis for continuous-time optimal control. Second, an integral-reinforcement-learning-based hybrid iterative algorithm is developed by combining the advantages of policy iteration and value iteration. The proposed algorithm improves the convergence performance of the optimal solution without requiring an admissible initial control policy or precise dynamic information. On this basis, a fuzzy logic system is employed to approximate the performance index function, and the fuzzy weights are updated using an experience replay mechanism, thereby further relaxing the requirement of persistent excitation conditions. Finally, Lyapunov theory demonstrates the error of closed-loop MRM system is uniformly ultimately bounded and visualization data of simulation and experimental reveal the validity of the proposed method.</p>