<p>Efficient and resilient control of water distribution systems (WDS) is critical for sustainable infrastructure management under increasingly uncertain demand conditions. This study presents a comprehensive benchmarking and sensitivity analysis of three reinforcement learning algorithms-Proximal Policy Optimization (PPO), Deep Q-Network (DQN), and Asynchronous Advantage Actor-Critic (A3C)-for water tank scheduling across multi-day planning horizons. Our simulation-based framework incorporates realistic demand variability, extreme operational scenarios, and temporal modeling using LSTM networks to enable robust agent training. Extensive evaluation reveals that PPO achieves superior performance in long-horizon scenarios with up to 40% fewer pump activations and 25% fewer safety violations than DQN, while maintaining competitive performance across shorter horizons. A detailed sensitivity analysis identifies learning rate as the most critical hyperparameter, with DQN showing narrow optimal ranges (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">$1\times 10^{-3}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>1</mn> <mo>×</mo> <msup> <mn>10</mn> <mrow> <mo>-</mo> <mn>3</mn> </mrow> </msup> </mrow> </math></EquationSource> </InlineEquation>) compared to PPO’s broader robustness (<InlineEquation ID="IEq2"> <EquationSource Format="TEX">$1\times 10^{-5}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>1</mn> <mo>×</mo> <msup> <mn>10</mn> <mrow> <mo>-</mo> <mn>5</mn> </mrow> </msup> </mrow> </math></EquationSource> </InlineEquation> to <InlineEquation ID="IEq3"> <EquationSource Format="TEX">$3\times 10^{-4}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>3</mn> <mo>×</mo> <msup> <mn>10</mn> <mrow> <mo>-</mo> <mn>4</mn> </mrow> </msup> </mrow> </math></EquationSource> </InlineEquation>). The ablation study demonstrates that while LSTM networks enhance temporal modeling, the simpler DQN-FFN architecture notably outperforms LSTM-augmented counterparts, achieving superior cumulative rewards (−93.85 vs −134.15 for PPO-LSTM). Under extreme demand noise up to ±50 units, PPO demonstrates exceptional robustness with only 12% performance degradation compared to 28% for DQN. The study provides practical guidelines for algorithm selection, hyperparameter tuning, and action-space design, establishing a foundation for transparent AI-driven control in complex WDS and directly implicating Industry 4.0/5.0 infrastructure modernization.</p>

Multi-algorithm reinforcement learning framework with feedforward networks for resilient water tank scheduling systems

期刊论文
出版模式：

开放获取
发表日期：

2026年4月8日

查看全文

Hee-Beom Park,
Akeem Bayo Kareem,
Yusuf Olatunji Kareem,
Habeeb Adewale Ajimotokan,
Jang-Wook Hur

摘要

Efficient and resilient control of water distribution systems (WDS) is critical for sustainable infrastructure management under increasingly uncertain demand conditions. This study presents a comprehensive benchmarking and sensitivity analysis of three reinforcement learning algorithms-Proximal Policy Optimization (PPO), Deep Q-Network (DQN), and Asynchronous Advantage Actor-Critic (A3C)-for water tank scheduling across multi-day planning horizons. Our simulation-based framework incorporates realistic demand variability, extreme operational scenarios, and temporal modeling using LSTM networks to enable robust agent training. Extensive evaluation reveals that PPO achieves superior performance in long-horizon scenarios with up to 40% fewer pump activations and 25% fewer safety violations than DQN, while maintaining competitive performance across shorter horizons. A detailed sensitivity analysis identifies learning rate as the most critical hyperparameter, with DQN showing narrow optimal ranges ( $1\times 10^{-3}$ $1 \times 10^{- 3}$ ) compared to PPO’s broader robustness ( $1\times 10^{-5}$ $1 \times 10^{- 5}$ to $3\times 10^{-4}$ $3 \times 10^{- 4}$ ). The ablation study demonstrates that while LSTM networks enhance temporal modeling, the simpler DQN-FFN architecture notably outperforms LSTM-augmented counterparts, achieving superior cumulative rewards (−93.85 vs −134.15 for PPO-LSTM). Under extreme demand noise up to ±50 units, PPO demonstrates exceptional robustness with only 12% performance degradation compared to 28% for DQN. The study provides practical guidelines for algorithm selection, hyperparameter tuning, and action-space design, establishing a foundation for transparent AI-driven control in complex WDS and directly implicating Industry 4.0/5.0 infrastructure modernization.