With the construction and grid integration of large-scale photovoltaic power generation systems, utilizing energy storage technology to reduce grid-connected power fluctuations and enhance grid stability has become a research hotspot. This paper, based on a hybrid energy storage system composed of flywheels and lithium-ion batteries, analyzes the measured photovoltaic output power, establishes a hybrid energy storage system model to smooth the fluctuation rate of photovoltaic power generation. Addressing the power allocation issue of the hybrid energy storage system, an optimization algorithm (Arithmetic Optimization Algorithm, AOA) combined with Variational Mode Decomposition (VMD) is employed to solve the model. The simulation results demonstrate that the AOA-VMD algorithm reduces the rated power of both lithium batteries and flywheels. This validates its superiority in smoothing photovoltaic output fluctuations and addressing the mode mixing problem of non-stationary signals, as well as its effectiveness in optimizing power allocation in hybrid energy storage systems. It enhances the efficiency and operational reliability of hybrid energy storage systems, thus advancing the large-scale application of renewable energy sources.