Trajectory similarity computation based on deep representation learning leverage latent representations for similarity calculation between trajectories. As an essential evaluation index and measurement standard, the vulnerability of trajectory similarity computation directly affects the safety of autonomous driving tasks. However, due to the strong discontinuity in the output of the trajectory representation model and the naturally unlabelled nature of trajectory data, robustness assessment is more difficult for trajectory representation models. In this work, we investigate the vulnerability of trajectory similarity computation for the first time, and define the adversarial criterion for trajectory representation models. We propose a decision-based attack framework against trajectory representation models, namely Rball Attack. Experiments on real-world datasets demonstrate that Rball Attack achieves high attack success rate in adversarial attacks on SOTA.