Gas–liquid swirling flow is critically important across various industries, including nuclear engineering. At high gas volume fractions, separation efficiency is significantly influenced by the hydrodynamic instability of gas–liquid swirling flow. Understanding the transient behavior of unstable swirling flow is essential for optimizing separation processes. This study employs high-speed imaging to capture dynamic characteristics of unstable swirling flows, and quantifies gas core diameter fluctuations quantified through advanced image processing. The results demonstrated that different inlet flow patterns generate distinct unstable gas–liquid swirling flows, which directly affect separation efficiency. Flow instability-induced partial phase mixing significantly degrades separation efficiency. Quantitative analysis demonstrates that the gas core diameter increases with gas phase velocity but decreases with elevated liquid phase velocity. The standard deviation of gas core diameter fluctuations initially rises and subsequently declines with increasing gas velocity. Time–frequency analysis further identifies a transition in diameter oscillation patterns: high-frequency, low-amplitude fluctuations dominate at low gas volume fractions, whereas low-frequency, high-amplitude oscillations prevail under high gas-loading conditions. These findings provide critical insights into the instability mechanisms of gas–liquid swirling flows.