<p>Gaze analysis provides insightful information in several areas, making it increasingly popular among researchers and practitioners. With the development of deep learning, appearance-based gaze estimation has attracted particular interest. However, research in this area is often limited to testing novel methods on identical popular backbones, such as ResNet or VGG. This study comprehensively evaluated 22 deep learning architectures across four benchmark datasets to establish a baseline for future research in appearance-based gaze estimation and to encourage researchers to use more efficient architectures to achieve new state-of-the-art. We proved that a carefully selected backbone can outperform sophisticated methods. We analyzed the gaze estimation accuracy for each architecture, considering the angular distance for the gaze vector and separately the errors in estimating individual components - pitch and yaw. We further examined the impact of architectural complexity on the results. Furthermore, we conducted a rigorous cross-dataset evaluation to assess the generalization capabilities of the architectures under significant domain shifts. Our analysis reveals critical limitations in current state-of-the-art backbones, including susceptibility to model collapse and poor extrapolation performance when transferring between screen-based and wide-angle gaze distributions. Finally, we combined explainable AI with a quantitative ablation study to confirm that the eye area is the dominant factor in gaze estimation. By comparing full-face and eyes-only inputs, we proved that other facial regions provide negligible information, validating the potential for optimizations through region-specific focus. Our findings provide valuable insights for researchers and practitioners, a reliable baseline for future studies, and guidance on selecting optimal architectures tailored to specific application requirements. The source code for this study is available upon reasonable request to the corresponding author.</p>