The investigation of convective heat transfer (CHT) with varying features of cavity is crucial for engineering applications where efficient heat transfer is essential. Building on this, the current study examines the effect of hybrid nanofluids in a cavity on mixed convection (MC), enhanced by the presence of a heated rotating cylinder. While the previous studies treat these geometries separately, the unique novelty of this work lies in examining the combined impact of a wavy boundary, corner heating, and a rotating cylinder under Buongiorno’s two-phase model. The cavity features a wavy wall maintained at a constant cold temperature ( \(T_{\text {c}}\) ), and on the opposite corner, is surrounded by a heated wall at a constant high temperature ( \(T_{\text {h}}\) ). The dimensionless governing equations, incorporating the two-phase Buongiorno’s model, are derived and subsequently solved using the GWR-FEM with geometry-specific boundary conditions. The derived mathematical model and its numerical solutions are cross-validated with the previous studies, showing excellent agreement. A parametric study is conducted to investigate the fluid behavior across various number of undulations (N = 1–4), angular rotational velocity ( \(\Omega =100,300,500\) ), and Rayleigh number ( \(\text {Ra}=10^3, 10^4, 10^5\) ). The Nusselt number ( \(\overline{\text {Nu}}\) ) obtained varies from approximately 3.8 to 5.2, showing that heat transfer increases by 31.11% when \(N=1\) , \(\text {Ra}=10^5\) , and \(\Omega =500\) , underscoring the significant impact of the analyzed parameters on CHT efficiency. Furthermore, the Bejan number ( \(\text {Be}\) ) varies significantly across the parametric space, reaching its peak ( \(\text {Be} = 0.8147\) ) at weak convective intensity ( \(\text {Ra} = 10^3\) ) and dropping to its minimum ( \(\text {Be} = 0.0034\) ) in the strongly convection-dominated regime ( \(\text {Ra} = 10^5, \Omega = 100, N = 4\) ), indicating a complete shift from heat transfer-dominated to fluid friction-dominated irreversibility. The insight from this study is crucial for optimizing thermal designs in applications where minimizing fluid friction losses is paramount, such as heat exchangers and cooling systems.