<p>Device-to-device communication is an emerging technology and the future of wireless networks. It has the capability to reduce the burden on the base station and make communication fast. In this paper, we have investigated the intricate dynamics of transmission capacities enhancement of underlay and overlay device-to-device (D2D) communication at a D2D link distance of 50&#xa0;m, focusing on their relationship with D2D transmission power, relay node transmission power, D2D density, potential RN density, signal-to-interference ratio (SIR) threshold of relay transmission, D2D link distance, etc. We have got several key results, focusing light on the nuanced interplay between these variables. It has been observed that at a link distance of 50&#xa0;m, as the D2D transmission power increases, the SIR threshold of relay transmission increases, and D2D transmission capacity decreases. This paper delves into the optimisation of the transmission capacities of underlay, overlay D2D communications in different situations. We have derived the expressions for optimum D2D density and computed the maximum value of transmission capacities for underlay and overlay communication at D2D transmission powers of 15 dBm and 30 dBm. The maximum values of transmission capacities at D2D transmission powers of 15 dBm have been found as 4.4 × 10^− 3 bps and 8.8 × 10^− 3 bps for the concerned underlay and overlay systems, respectively, which is better as compared to previous research[4]. In addition, the energy efficiency of both systems has been analyzed and found the overlay system to be better.</p>