<p>Magnetic nanofluids with unique thermophysical properties can serve as potential coolants in heat exchanger systems owing to their tunable magnetic response. The study aims at enhancing the thermal efficiency and flow dynamics in shell and helically coiled tube heat exchanger with nanofluids and hybrid nanofluid using the multiphase mixture computational fluid dynamics approach. The numerical results are validated against findings from the experimental analysis. Spinel-type manganese ferrite (MnFe₂O₄) nanofluid having superparamagnetic nature and graphene oxide (GO) nanofluid are considered for this study. The hybridization and synergetic effect of MnFe₂O₄ and GO nanostructures give rise to enhanced magnetic properties and improve the thermal performance owing to relatively greater thermal conductivity of graphene oxide. Four volume fractions (0.1 vol%, 0.5 vol%, 0.75 vol%, and 1 vol%) of nanoparticles are investigated, and results indicate that the thermal efficiency enhances with an increase in volume fraction. The ratio of the nanoparticles in hybrid nanofluid is optimized to 70:30 with MnFe₂O₄ to GO nanostructures. Compared to water, the heat transfer rate, overall HTC, and pressure drop enhanced up to ~85%, ~185%, and ~68%, respectively, while employing MnFe₂O₄-GO/water hybrid magnetic nanofluid. Moreover, the application of magnetic field with intensity 1000 G increases the heat transfer rate, and thereby, the effectiveness of the heat exchanger reaches its maximum of 0.79, which is 88% higher than water. The velocity contours demonstrate on how the magnetic field influences and controls the flow of the magnetic nanofluid.</p> Graphical Abstract <p></p>