<p>Efficient thermal management in confined geometries is critical for advancing electronics, solar energy systems, and biomedical platforms. This study presents a numerical investigation of convective heat transfer enhancement in a finned square cavity filled with Cu–water nanofluid and subjected to a transverse magnetic field. The cavity, with an aspect ratio of 3, incorporates three internal vertical fins (length 0.5) positioned at 40% of the cavity height along the heated wall. Simulations were performed using ANSYS Fluent 15.0, employing the finite volume method, SIMPLE algorithm, and the Boussinesq approximation to account for buoyancy-driven flow. A parametric analysis evaluates the influence of Rayleigh number (<i>Ra</i>), Hartmann number (<i>Ha</i>), and nanoparticle volume fraction (<i>φ</i>) on thermal performance. At <i>Ra</i> = 10⁶ and <i>Ha</i> = 10, the configuration yields a maximum increase of 32% in the mean Nusselt number on the cold wall relative to the non-magnetic nanofluid case. The effect of nanoparticles is found to be <i>Ra</i>-dependent, with negligible impact at low <i>Ra</i> and significant enhancement at higher <i>Ra</i> due to intensified thermal boundary layer disruption. Increasing <i>Ha</i> compresses the flow domain and shifts the regime toward conduction dominance, thereby reducing convective strength. Streamline and isotherm contours reveal that fin-induced flow modulation interacts with magnetic damping to restructure heat transfer pathways. The integrated treatment of magnetic field effects and nanoparticle-enhanced convection demonstrates the potential for developing magnetically tunable enclosures and passive cooling systems. The findings offer design insights applicable to high-performance heat exchangers, energy storage units, thermal mixing chambers, and reactor components.</p> Graphical abstract <p></p>