<p>Z-type hexaferrite nanoparticles, Ba₃Co₂In_0.1La_xFe_23.9−xO₄₁ (<i>x</i> = 0.00, 0.05, 0.1, 0.15, and 0.2), were synthesized using the auto-combustion sol–gel technique. These nanoparticles have the potential to decrease the size of antennas utilized in microwave and ultra-high frequency communication. Substituting La³⁺ cations in the Co₂Z hexaferrite structure improved the magnetic and electromagnetic properties. The authenticity of the results was confirmed through X-ray Diffraction Pattern (XRD), Scanning Electron Microscope (SEM), and Energy Dispersive X-ray Analysis (EDAX). The XRD measurements revealed that the samples were crystalline, forming a pure Z-phase with the space group P63/mmc with particle size 15.11 nm to 15.94 nm, with theoretical density ranging from (5.24–5.48 g/cm³). Magnetic studies conducted using a Vibrating Sample Magnetometer (VSM) demonstrated an anomalous increase in saturation magnetization from 65.93 emu/g for the undoped sample to 110.82 emu/g with x = 0.2 doping, low retentivity values for all doped samples except Ba₃Co₂In_0.1La_0.2Fe_23.7O₄₁ and low coercivity except for pure and Ba₃Co₂In_0.1La_xFe_23.9−xO₄₁ samples. The values of the anisotropy field range from (2.1–2.97 kOe). The electromagnetic characteristics were analyzed using a Vector Network Analyzer (VNA). The sample with pure composition and 0.2 La doping exhibited the best overall performance across the frequency range of 1–8 GHz, demonstrating remarkably low dielectric and magnetic loss. The real permittivity values increased from 7 to 10.6 with lanthanum doping (<i>x</i> = 0.0 to <i>x</i> = 0.2), with losses ranging between 0.1 and 0.3. Loss tangent graphs and reflection loss graphs were also presented further to clarify the energy parameters of the electromagnetic properties.</p> Graphical Abstract <p></p>