This work delves into an advanced numerical analysis and comparative investigation of two distinct radiative Casson hybrid nanofluids, \({\text{H}}_{{2}} {\text{O}} - {\text{Ag}} - {\text{MgO}}\) and \({\text{H}}_{{2}} {\text{O}} - {\text{Cu}} - {\text{Al}}_{{2}} {\text{O}}_{{3}}\) , flowing over a rotating cone. Sophisticated computational Runge–Kutta method and shooting procedure are employed to thoroughly evaluate the effects of exponential space dependent heat source (ESHS) parameter, Maxwell velocity slip condition, Smoluchowski temperature slip condition, mixed convection parameter, volume concentration, cone’s half-vertex angle, exponential index, Casson parameter, magnetic parameter, radiation parameter and Prandtl number. The outcomes reveal that \({\text{H}}_{{2}} {\text{O}} - {\text{Ag}} - {\text{MgO}}\) exhibits improved tangential velocity profiles, while \({\text{H}}_{{2}} {\text{O}} - {\text{Cu}} - {\text{Al}}_{{2}} {\text{O}}_{{3}}\) shows better temperature distribution profiles. Both nanofluids demonstrate similar behavior in their swirl velocity profiles around the cone's surface. Three independent factors, volume fraction, radiation parameter, and temperature slip condition, are examined using the response surface methodology—central composite design model (RSM–CCD) in order to maximize the heat transfer rate. The findings indicate that enhancing the radiation parameter significantly improves heat transfer, highlighting the importance of optimizing the system's thermal performance. Sensitivity analysis reveals that the Nusselt number reaches to peak when the radiation parameter is constant, volume concentration is minimal, and the temperature slip condition is maximized. The sensitivity of heat transfer rate to \(\partial_{{\gamma_{2} }} Nu\) decreases as \(\phi :0.01 \to 0.05,\) \(\gamma_{2} = 0.2 \to 0.6\) varies from minimum to maximum values when \(Rd\) is held at 1.5, with the highest sensitivity value (−0.511035) observed at the uncoded values \(\phi = 0.05,\,Rd = 1.5,\gamma_{2} = 0.2\) and the least value (−0.666383) noted at the uncoded values \(\phi = 0.01,\,Rd = 1.5,\gamma_{2} = 0.6\) . This approach identifies the factors affecting thermal efficiency and optimizes system performance across various fluidic and radiative scenarios. The study renders valuable insights for enhancing heat transmission efficiency in hybrid nanofluid applications, particularly in systems requiring precise thermal control in rotating and radiative conditions, such as gas turbines, high-speed rotating machinery, advanced automotive cooling systems, and solar thermal energy collectors.
