Clinical time series are ubiquitous in healthcare for accurate disease risk prediction. The recent Transformer models have demonstrated superior performance in time series learning. However, these methods focus on global temporal dependency and widely utilize channel-mix or channel-independent tokenization. They ignore local dependency in clinical time series and are limited to capturing the intrinsic clinical variable correlations. To address the above issues, we present an Multi-scale Value-Density Transformer with Medical Semantic Guidance (MVMformer), which takes irregularity-aware segment-wise modeling for clinical time series and correlate diverse clinical variates based on their medical semantic affinity. Specifically, MVMformer introduces a Multi-scale Value-Density Attention to capture intra-segment characteristics in both value trends and temporal density while accommodating multi-length segments. Furthermore, MVMformer constructs a Hierarchical Medical Semantic Graph to analyze complicated variable relationships starting from detailed measurements to associated organs. Experimental results on three medical datasets demonstrate the superiority of MVMformer over existing state-of-the-art methods.