Positron emission tomography (PET) reconstruction is a challenging inverse problem, where projection data often contain low statistics. While current supervised learning methods offer strong noise suppression abilities, they may suffer from generalization issues and are in many cases not accurate quantitatively. To overcome these challenges, we propose a novel self-supervised Time-of-Flight PET (TOF-PET) reconstruction framework that utilizes Implicit Neural Representations (INR) to model PET images. Specifically, we introduce a differentiable forward projection model based on the imaging mechanism for TOF-PET and reformulate TOF-PET reconstruction problem using INR. To enhance image smoothness, we develop a ray-based total variation (TV) regularization term, distinct from the traditional TV. For the internal structure of our INR, we integrate a multi-resolution hash encoder with our designed prior-image encoder, where the latter provides sufficient image prior and always delivers reliable initial reconstructions for arbitrary network depth. Experiments on brain and chest datasets show that our method outperforms traditional iterative algorithms and self-supervised approaches in noise suppression and contrast recovery. Compared to conventional NeRF-based architectures, our model is more compact and converges faster, providing an efficient solution for TOF-PET reconstruction. The source code repository is hosted on GitHub: https://github.com/zyl123300/PD-INR.git .