<p>The phonon density of states (PhDOS) governs the lattice vibrational and thermodynamic properties of numerous crystalline materials. Traditional approaches to obtaining the PhDOS, including experimental measurements and density functional theory calculations, are often time-consuming and expensive. In contrast, machine learning provides an efficient alternative for accurately predicting phonon spectra and PhDOS, especially for complex and exotic systems. In this work, we propose the Pre-trained Phonon Transformer (PPhT), an efficient PhDOS prediction model based on the attention mechanism. For a given crystal with element information, atomic positions and lattice information, PPhT encodes the input information and feeds the embeddings into the attention scoring mechanism, thereby enhancing its geometric awareness and enabling parallel modeling of atomic interactions. PPhT provides a computationally efficient alternative for mapping atomic configurations to vibrational dynamics by the Transformer-based network.</p>