Phylogenetic distance estimation and distance-based phylogeny reconstruction are well-studied cornerstone topics in phylogenetics. Classical approaches for both utilize mathematical or probabilistic graphical models of biomolecular sequence evolution. But model violations can occur and model-based analysis can be impacted as a result. Recent advances in statistical machine learning using deep neural networks provide an alternative in the form of representation learning. Newer applications of deep learning to phylogenetic distance estimation have followed. A number of challenges in this area remain, since state-of-the-art methods are often restricted to pairwise or subset-based analyses and retain other simplifying assumptions. In fact, classical model-based methods and representation learning are orthogonal and, as we show, their combination can be greater than the sum of its parts. We bridge these different approaches by synthesizing mathematical and logical constraints with statistical machine learning – an approach from physics-informed machine learning (PIML). Our algorithmic solution takes the form of a Transformer-based framework for learning phylogeny-informed representations directly from MSAs, which we apply to the task of phylogenetic distance estimation. The result is FIREFLY, a computational framework for “PHYlogeny-informed REpresentation learning to estimate PHyLogenetic dIstances”). We benchmarked FIREFLY’s performance against other state-of-the-art methods using simulated and empirical datasets. We found that FIREFLY improves both pairwise distance estimation accuracy and downstream phylogenetic inference compared with state-of-the-art methods. The gains are particularly pronounced under high indel rates and on estimated MSAs, where alignment errors and gap-induced uncertainty are most severe. Our results highlight the value of integrating phylogeny-based inductive bias into deep representation learning and suggest that MSA-level modeling offers a robust foundation for evolutionary inference under challenging conditions.