Point-supervised temporal action localization (P-TAL) aims to localize actions in untrimmed videos with only one action-level timestamp annotation. Owing to label sparsity in P-TAL, most existing methods locate action by boundary-free considering classification, which results in incomplete action localization. In this paper, we propose a novel point-to-boundary hierarchical learning framework that generates dense and reliable pseudo-action proposals to provide complete action annotation containing boundaries for a multi-granularity localization model with single-stage inference. Concretely, we first achieve action proposal boundary enhancement which takes the annotated points as corresponding action centers to search for the boundary-accurate action proposals by fitting a Gaussian prior according to Class Activation Sequences. Furthermore, to more noise-resiliently learn boundary information of actions from the obtained proposals, we introduce a hierarchical boundary regression algorithm and a multi-level attention loss function. Extensive experimental results on three challenging benchmarks demonstrate the state-of-the-art performance and the generalization of the proposed framework. Notably, our method even shows faster single-stage inference with better performance compared to the existing methods.