Human Action Recognition (HAR) has recently achieved significant success through the analysis of human behavior using non-visual data (e.g., sensor data) and visual data (e.g., skeleton data). However, sensor-based methods face challenges due to the inherent limitations of sensor data, including the absence of 3D body pose information, high volatility, and vulnerability to noise. Meanwhile, skeleton-based methods, while effective due to their rich spatial and temporal information, are constrained by the stringent requirements for data acquisition and problems like occlusion, limiting their feasibility in real-world outdoor scenarios. Therefore, to solve these challenges, we resort to the cross-modal generation strategy and aim to generate hard-to-collect but information-rich skeleton data conditioned on easy-to-monitor sensor data. In our work, we propose a novel Sensor-to-Skeleton Diffusion Model with Lipschitz Regularization, named SSDL. Specifically, we first design an Angular Variation module and extract angular variation information of joint movements with time information. Subsequently, noise is added to the skeleton key points and angular variation during the forward diffusion process. To address noisy sensor data and improve training stability, we incorporate Lipschitz regularization with the diffusion model’s loss to prevent overfitting. We verify the generalizability and effectiveness of our methods on two benchmark multimodal human action datasets: UTD-MHAD, Berkeley-MHAD, and SmartFall-MHAD dataset. Extensive results demonstrate the superiority of leveraging generated skeleton information conditioned on the sensor data for accurate human activity recognition with limited computational demands. Code will be available at https://github.com/nikhiliit/SSDL .