This study investigates the shielding, structural, linear, and nonlinear optical properties of a 50B2O3-20SrO-20Na2O- (10-x)PbO-xHgO (0 \(\le\) x \(\le\) 10.0 mol%) glass system prepared via the melt-quenching technique. Increasing HgO doping leads to a decrease in density from 3.87 to 2.62 g/cm3 and a corresponding expansion in molar volume from 24.34 to 29.28 cm3/mol. Optical analysis revealed a significant red shift in the absorption edge, with the indirect optical energy gap (Eg) decreasing from 3.0891 eV (x = 0) to a minimum of 2.9733 eV (x = 2), while the Urbach energy (EU) exhibited a marked increase, peaking at 0.4734 eV. The refractive index (n) peaked at 2.4043 for x = 2, correlating with the highest electronic polarizability (αe) and permittivity (ε). The metallization criterion (M ~ 0.39) confirmed the insulating nature of the samples. The HgO-doped borate glasses are excellent candidates for UV-shielding, high-index optical lenses, and non-linear photonic applications such as laser hosts and optical fibers. Mass attenuation coefficient (MAC), the photon-energy dependence of the mean free path (MFP) and half-value layer (HVL), the effective atomic number ( \({\text{Z}}_{eff}\) ), and electronic cross section (ECS) of the proposed glasses were evaluated. Results proved that the substitution of PbO with HgO leads to a consistent reduction in the combined shielding performance across all evaluated descriptors. Therefore, the proposed glasses HgO-10 is the best ones as gamma-ray protection.