In this paper we show how leveraging the structure of the solution space of the Traveling Salesman Problem (TSP) can lead to a dramatic improvement of the performance of state of the art diffusion-based neural solvers. Building on recent approaches of DIFUSCO and T2TCO which pipeline a diffusion-based solution generation with a local search procedure, we propose IDEQ (constrained Inverse Diffusion and EQuivalence class-based training of diffusion models for combinatorial optimization). IDEQ improves the quality of the solutions by leveraging the constrained structure of the TSP state space. Indeed, the solution space consists of locally optimal Hamiltonian tours which is a much smaller space than the space of adjacency matrices used in previous works. Also, the local search procedure defines an equivalence class of Hamiltonian tours: all elements of this equivalence class reach the same local optimum after the application of the local search. This should be aligned with the supervised training objective of the diffusion. IDEQ addresses these two points. Our experiments show that IDEQ achieves 0.3% to 0.4% optimality gap on TSP instances made of 500 cities, and 0.5% to 0.6% optimality gap on TSP instances with 1000 cities. This sets a new SOTA for neural based methods solving the TSP. IDEQ also performs well on the instances of the TSPlib, a reference benchmark in the TSP community, outside of the training distribution, with optimality gaps ranging from 0.9 to 1.1 %.