A series of quaternary borate-based glasses with the composition [x \({\text{SnO}}_{2}\) –(80–x) \({\text{B}}_{2}{\text{O}}_{3}\) –19.2 \({\text{Na}}_{2}\text{O}\) –0.8NiO] (x = 0–4 mol%) were synthesized via the melt-quenching method. The article evaluates the influence of \({\text{SnO}}_{2}\) incorporation of structure and optical properties. XRD confirmed the amorphous glass structure, with a complete absence of crystalline phases. The density increased from 2.16 g.cm−3 (for x = 0) to 2.54 g.cm−3 (for x = 4), while the molar volume decreased from 31.51 cm3.mol⁻1 to 28.17 cm3.mol−1, confirming network crosslinking and compaction. Optical absorption spectra exhibited Ni2+ transitions at ~ 421, 474, 693, 767, and 870 nm, with ligand field splitting energy (10Dq) increasing, while Racah parameter (B) decreased with \({\text{SnO}}_{2}\) content, suggesting that the bond nature is more stronger covalent. The optical band gap narrowed from 3.60 eV (for x = 0) to 3.35 eV (for x = 4). Correspondingly, the refractive index increased from 2.27 to 2.33, nonlinear refractive index ( \({n}_{2}\) ) enhanced from 0.61 × 10−13 to 0.71 × 10−13 esu, and third-order susceptibility (χ(3)) augmented from 1.01 × 10−13 to 1.19 × 10−13 esu. These findings demonstrate that \({\text{SnO}}_{2}\) -modified Ni2+-doped borate glasses exhibit enhanced structural compactness and nonlinear optical responses, making them promising candidates for photonic and optoelectronic applications.