This study investigates the influence of hybrid nanofluid parameters on surface roughness in the milling of hardened SKD61 tool steel. The cooling/lubrication system employed a combination of Al2O3 and SiO2 nanoparticles dispersed in a base fluid of coconut oil, with variations in nanoparticle ratio (A: Al2O3 concentration; B: SiO2 concentration), flow rate (F), and pressure (P). A total of 27 experimental runs were designed and performed using Taguchi approach. The results reveal that both the type and ratio of nanoparticles significantly affect surface roughness (Ra). Specifically, increasing the SiO2 concentration (B) while maintaining moderate Al2O3 levels (A) led to improved surface finish. Among all trials, the lowest surface roughness (Ra = 0.23 µm) was achieved at A = 2.0, B = 0.5–1.0, P = 3–5 bar, and F = 120 ml/min. In contrast, runs with A = 0.0 and B = 0.0 underperformed, resulting in the highest Ra values (up to 0.38 µm), confirming the enhancement effect of the hybrid nanofluid. Overall, the findings suggest that an optimized hybrid nanofluid composition and delivery parameters can significantly improve surface quality in hard steel machining. The results provide a basis for selecting effective nano-cooling conditions in sustainable and high-precision milling of hardened tool steels.