It has been an intriguing but challenging topic to achieve safe time-varying formation control using local information for a long time. In this paper, we investigate the problem of controlling multiple unmanned aerial vehicles (UAVs) modeled by a directed communication graph to follow a leader in formation, supported by a vector field-based collision avoidance strategy and a relative localization algorithm (RLA) by distance and velocity measurements. The proposed method can realize the safe time-varying formation flight with different patterns, and also waives the need of external localization systems. The relative localization algorithm is designed based on the recursive least square estimation (RLSE) technique with a forgetting factor. We present a specific protocol for the design of reference relative velocity in formation flight according to the desired formation pattern. The reference relative velocity can ensure the persistent excitation that ensures the convergence of the relative localization algorithm by RLSE. By assuming that UAVs have an omnidirectional range perception ability, it is shown that collision avoidance can be achieved using a vector field method by sensing the outline of surrounding obstacles or neighbour UAVs. The competence of the proposed algorithm is demonstrated via both numerical simulations and experiments.