This paper presents a new approach, including deep neural networks for adaptive modulation selection in free-space optical (FSO) communication systems under the mixed FSO/MIMO-MMW design framework for 5G and 6G networks. Adaptive modulation selects the best modulation scheme based on the channel conditions (atmospheric turbulence) to improve the bit error rate (BER) and transmission capacity. This article presents two research strands targeted at improving mixed FSO/MIMO-MMW communication systems. In the first strand, we model the experimentally measured refractive index structure parameter ( \({C}_{n}^{2}\) ) by conducting a comprehensive comparison of four machine learning regression algorithms: K-Nearest Neighbours, extreme Gradient Boosting, Artificial Neural Networks, and Deep Neural Networks. In the second strand, we employ DNN-based classification to apply adaptive modulation to the FSO link, enabling real-time evaluation of atmospheric conditions and dynamic selection of the most suitable modulation scheme. The switching threshold of modulation is set at \({\sigma }_{R}^{2 }=1\) , in which the system prefers 8-QAM under weak turbulence regimes ( \({\sigma }_{R}^{2 }< 1\) ) for optimum transmission capacity, and shifts to OOK under strong turbulence ( \({\sigma }_{R}^{2 }> 1\) ) to support strong, low-BER transmission. We also analyze the BER and channel capacity of the mixed FSO/MIMO-MMW system. The adaptive modulation scheme realizes an all-time best trade-off between reliability and capacity, significantly enhancing the robustness and efficiency of the mixed system when exposed to weather turbulence.
