<p>Waste heat carried by coal mine return air represents an important form of low grade geothermal energy, and its efficient utilization is closely related to mine thermal hazard control and geothermal energy management. To improve the coupled heat and mass transfer performance in coal mine return air waste heat recovery systems, this study proposes an optimization and evaluation method based on entransy dissipation theory. A counter current air–water heat and mass transfer model in a vertical spray chamber is established, incorporating droplet dynamics and thermodynamic coupling. On this basis, a dimensionless indicator, namely entransy dissipation conversion efficiency, is introduced to characterize the proximity of the actual heat and mass transfer process to the ideal reversible state. The effects of key operating parameters—including spray chamber height, droplet diameter, and initial droplet velocity—on system performance are systematically analyzed and compared with conventional indicators such as enthalpy efficiency and exergy efficiency. The results show that when the total entransy dissipation of the system approaches zero, the entransy dissipation conversion efficiency reaches unity, indicating an optimal heat and mass transfer process. Compared with traditional evaluation metrics, the proposed indicator provides a clearer physical interpretation of the irreversibility mechanism and a more effective criterion for system optimization. This study offers a new theoretical tool for the efficient utilization of coal mine return air geothermal energy and the optimization of mine waste heat recovery systems.</p>