In recent years, significant progress has been made in improving the efficacy of wearable human activity recognition (HAR) tasks using deep learning technology. Existing research indicates that stacked convolutional layers effectively extract high-semantic signal features from multi-sensor channel time-series data. However, these approaches disregard the fact that sensor signals are low-semantic and sensor-based deep networks lead to overfitting and gradient vanishing. In this paper, we present the parallel attention-based HAR (PA-HAR) method. Our method employs multiple small-scale receptive fields to extract low-semantic signals in parallel and a skip-squeeze excitation block to establish correlations among multi-feature maps based on the feature channel dimension. We also introduce smooth and non-monotonic sigmoid linear units (SiLU) to integrate multi-scale and cross-channel features in order to prevent the loss of non-linear information due to small-scale receptive fields and reduce representational ability loss. Extensive experiments on seven public datasets show that our proposed PA-HAR model outperforms state-of-the-art approaches in HAR tasks. In addition, we develop a wearable real-time activity recognition system based on the embedded device with our model.