Exchange-Traded Funds (ETFs) are financial instruments that require periodic rebalancing to maintain their target asset allocations. However, the process of index composition changes exposes ETFs to the risk of front-running, where traders preemptively trade in anticipation of large ETF orders, leading to increased transaction costs and diminished portfolio performance. This paper tackles the challenge of ETF rebalancing under index composition changes, while also considering the impact of front-running, by proposing a novel Reinforcement Learning (RL) framework. The objective is to complete the index composition changes while maximizing returns through reinforcement learning. Our experiments primarily use the Deep Deterministic Policy Gradient (DDPG) algorithm, with additional testing conducted using the Soft Actor-Critic (SAC) and Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithms. These methods utilize both sequential data, such as historical prices and technical indicators, and non-sequential data to optimize portfolio value during rebalancing. The research specifically focuses on the recently issued Yuanta Taiwan Value High Dividend ETF (00940), which encountered significant front-running during its public offering and portfolio construction. Our results demonstrate that the RL-based strategies not only improve portfolio returns but also mitigate the impact of front-running. This study contributes to the field by pioneering the application of RL in portfolio management and rebalancing over an extended time horizon of intraday trading data.