Traversing narrow spaces while ensuring formation stability and safety is a complex task for multi-agent systems (MASs). This paper proposes a novel control strategy to enable MASs to traverse narrow spaces in various formations adaptively. Within a leader-follower framework, V-shaped and regular polygonal formations are specifically designed for traversing narrow spaces. A modified behavior-based adaptive formation control strategy is presented to achieve this adaptive traversal. This strategy utilizes artificial potential fields (APFs) to ensure safety, while feedback control facilitates stability. The stability and safety are rigorously analyzed by using the Control Lyapunov Function (CLF) and the Control Barrier Function (CBF), respectively. Furthermore, theoretical analysis provides the conditions for guaranteed stability. The effectiveness of the proposed strategy in guiding MASs through narrow spaces in V-shaped or regular pentagon formations is validated through two simulations.