Traffic prediction is critical for Intelligent Transportation Systems (ITS), but limited available data across cities constrains predictive accuracy. Transfer learning is widely adopted to mitigate data sparsity from resource-rich cities, yet such cross-city predictions face challenges from semantic-topological misalignment and spatial granularity inflexibility. Due to the very coupling of semantic features with topological structures, existing methods are difficult to handle cross-domain variations effectively. To address these issues, we propose TASA, a transfer framework that employs semantic-topological decoupling mechanism and adaptive enhancement mechanism for multi-scale spatial attention. Through a collaborative training strategy in three stages, TASA can independently model transferable traffic semantics and city-specific topological structures; while through hierarchical block attention, it can dynamically adjust spatial granularity. Experiments on four real-world datasets demonstrate that TASA outperforms state-of-the-art baselines, achieving average improvements of 12.2% in MAE and 14.3% in RMSE, proving its effectiveness in cross-city traffic prediction of few-shot scenarios.