This paper presents the performance analysis of a fixed-gain amplify-and-forward dual-hop mixed radio frequency (RF)/free-space optics (FSO) communication system. To ensure broad applicability, the RF link is modeled using the \(\kappa -\mu\) fading distribution, while the FSO link is characterized by the unified \(\mathcal {M}\) -distribution, which accounts for atmospheric turbulence and pointing errors. The FSO receiver employs both intensity modulation with direct detection and heterodyne detection techniques. In this study, we derive closed-form expressions for the average symbol error rate (ASER) of higher-order quadrature amplitude modulation (QAM) schemes, particularly coherent rectangular QAM and coherent hexagonal QAM, in terms of the Meijer-G function. Furthermore, the ASER for non-coherent modulation schemes, such as binary frequency shift keying and differential phase shift keying, is analyzed by deriving the moment-generating function of the end-to-end signal-to-noise ratio. To validate the analytical expressions, the numerical results are compared with Monte Carlo simulation results. Additionally, a comprehensive comparative study of different modulation schemes is conducted, highlighting the impact of pointing errors, \(\kappa -\mu\) fading parameters, and atmospheric turbulence on the overall system performance.
