Infrared multi-object tracking (MOT) is essential for applications such as wildlife monitoring and nighttime surveillance. To address the problems of low robustness in infrared tracking and the scarcity of high-quality datasets, we propose a novel strategy that integrates infrared temperature characteristics with visual appearance information. Specifically, we introduce a temperature-guided attention module based on physical inversion. It leverages a remote-sensing radiance model to estimate surface temperature from brightness temperature, guiding the encoder to focus on thermally salient regions and enhancing the representation of low-texture targets. Furthermore, we propose a temporal heat conduction mechanism to enhance the temporal consistency of thermal features, ensuring more stable association across frames. In addition, we design a temperature-aware association method (TAM), which dynamically constructs a matching weight matrix by integrating temperature scores, trajectory uncertainty, and appearance similarity, aiming to improve the association stability in scenarios with occlusions and appearance ambiguity. Moreover, we construct a high-quality infrared wildlife MOT dataset IRWT, and standardize the formats of two existing infrared MOT datasets. Comparisons with five state-of-the-art MOT methods demonstrate that our approach achieves leading performance on the IRWT dataset. Evaluations on two additional non-animal infrared datasets further confirm its generalization and practical applicability.