Automated document classification is a vital research domain, utilizing approaches that employ textual features (e.g., plain text analysis), visual features (e.g., feature extraction from document images), or hybrid techniques that combine both modalities to capture both textual and visual features to enhance performance. Our work introduces two hybrid models, HEADoC \(_{BASE}\) with 27.7 million parameters and HEADoC \(_{LARGE}\) with 90.58 million parameters. A key innovation is our deep attention mechanism, a streamlined yet efficient method inspired by conventional attention frameworks, which facilitates the smooth integration of the two modalities. During experimentation, we observed that ArcFace loss, a metric learning approach effective for heterogeneous datasets with distinct intra-class characteristics, performed poorly in our task due to the homogeneity of document classes in standard benchmarks. Both models were evaluated on the RVL-CDIP and Tobacco3482 datasets. On Tobacco3482, HEADoC \(_{BASE}\) achieved 95.98% accuracy, while HEADoC \(_{LARGE}\) reached 96.66%. For RVL-CDIP, accuracies were 92.95% and 93.62%, respectively. HEADoC \(_{BASE}\) not only surpassed numerous state-of-the-art models but also proved itself to be the most compact architecture in comparison, highlighting efficiency without sacrificing performance. Having the most compact sizes amongst their competitors, our models can be trained faster than its rival architectures.
