<p>Heat exchangers play a critical role in improving energy efficiency in industrial thermal systems; however, simultaneously enhancing heat transfer while minimizing hydraulic losses remains a significant challenge. In this study, a new helical heat exchanger with a variable cross-sectional diameter is proposed, and the combined effects of geometric optimization and a water-based CuO + Al₂O₃ + TiO₂ ternary nanofluid on thermo-hydraulic performance are investigated. The thermophysical properties of the ternary nanofluid were experimentally measured and implemented in a validated three-dimensional CFD model. The results revealed that optimizing the conical diameter significantly reduced pressure drop and pumping power. Compared with the conventional 8.3 → 8.3&#xa0;mm configuration, the optimized geometry reduced pressure drop by up to 66.6% and increased the heat transfer-to-pumping power ratio by approximately 194%, while maintaining nearly the same heat transfer performance. Furthermore, the use of the ternary nanofluid increased the heat transfer rate from approximately 1750–2450&#xa0;W, corresponding to an enhancement of about 40%, while simultaneously reducing pumping power by nearly 45%. The findings demonstrate that geometric optimization primarily improves hydraulic performance, whereas the ternary nanofluid substantially enhances thermal performance. The proposed combined enhancement strategy provides an effective pathway for developing compact, energy-efficient, and high-performance helical heat exchangers. The study also reports the first application of a CuO + Al₂O₃ + TiO₂ ternary nanofluid in a conical helical heat exchanger, offering new insights into advanced thermal management systems.</p>