This study presents a novel comparative investigation of Ni- and Ag-doped \(Ca{Al}_{2}{O}_{4}-ZnO/rGO\) ternary composite thin films synthesized through a simple, scalable, and eco-friendly electrodeposition process. The work explores how dopant concentration modulates both linear and nonlinear optical properties, linking them to the electronic and structural behavior relevant for electrode applications. The optical bandgap varied from 2.37 eV (undoped) to 2.50 eV (Ni-doped, 0.06 M) thin film, indicating a Moss–Burstein shift and enhanced carrier concentration, while Ag doping narrowed the bandgap significantly from 2.37 eV to 1.63 eV (Ag-doped, 0.03 M), attributed to plasmonic resonance and defect-induced states. The Urbach energy (E₍ᵤ₎) increased from 1.52 eV (undoped) to 4.80 eV (0.03 M Ag), suggesting rising structural and electronic disorder. Concurrently, the absorption edge (Ed) expanded from 0.92 eV in the pristine film to 1.65 eV in highly Ag-doped samples, confirming defect-tuned optical transitions. Analysis of dielectric and nonlinear parameters revealed the real dielectric constant (ε′) exhibited negative values across all samples, signifying plasmonic-type behavior and free-carrier dominance. The high-frequency dielectric constant ( \({\varepsilon}_{\infty }\) ) ranged from 0.02 to 0.11, while the static refractive index (n₀) decreased from 0.33 to 0.14 as Ag concentration increased, indicating improved optical transparency. The linear optical susceptibility (χ1) decreased from − 0.68 to − 0.77, and the third-order nonlinear optical susceptibility (χ3) increased from 3.6 × 10⁻11 to 5.9 × 10⁻11, demonstrating higher electronic delocalization and nonlinear optical response, particularly in Ag-doped films. Correspondingly, the nonlinear refractive index (n2) fluctuated between 1.2 × 10⁻9 and 8.6 × 10⁻9, consistent with enhanced field-induced polarization. Overall, Ni doping primarily improved bandgap broadening, optical conductivity, and dielectric constant, signifying stronger electron–phonon coupling and higher charge storage capacity—key for supercapacitor electrodes—whereas Ag doping enhanced plasmonic response, χ3, and carbon cluster density, making it a promising candidate for nonlinear photonic and optoelectronic applications. This quantitative comparative analysis provides a novel understanding of dopant-induced multiphase interactions in ternary oxide–graphene composites and their significance in energy storage and optical device design.