Existing large-scale batteries, such as batteries for electric vehicles, electric planes, and electric boats/ferries, are designed so that battery cells are assembled by a combination of fixed parallel and series connections, making them unsuitable for efficient battery power management to improve the overall battery operating time. This paper proposes dynamic multi-cell battery reconfiguration based on deep reinforcement learning to prolong battery operating life. Specifically, unlike existing research mainly focused on cell-level battery reconfiguration, our approach directly optimizes the battery topology at the switch level while considering different battery and load constraints, explicitly generating the optimal topology for the entire battery pack. We use PyBaMM, an open-source battery simulation toolkit, to model multi-cell battery discharge behaviors in the reconfiguration optimization. Simulation results show that the proposed approach can significantly improve battery operating time while meeting various operational constraints.