Line-level handwritten text recognition (HTR) transcribes text from scanned or photographed documents, with transformer-based models recently achieving strong performance. However, these models typically require extensive real and synthetic datasets and significant computational resources, rendering them impractical for tasks involving small, private datasets. The lack of open-source additional data further hinders fair performance comparisons. This work proposes a Dual Downsample Vision Transformer (DDViT) for HTR, relying solely on small, open-source datasets like IAM. Building on an efficient vision transformer baseline, DDViT generates two predictions via distinct downsampling layers, selecting the final output with a novel dual maps scoring method. DDViT achieves state-of-the-art character error rates (e.g., 3.34% on IAM-A, a 1.5% point improvement over prior works) across nine related studies and remains competitive with methods using additional data, demonstrating effective HTR without resource-intensive requirements.