Human activity recognition (HAR) leverages data from wearable sensors or environmental devices to classify human activities. While deep learning models have significantly improved HAR accuracy, their complex architectures often lead to high computational cost and latency, limiting their applicability on edge devices. Lightweight models address this issue by reducing model size and complexity. However, they often compromise accuracy, especially for minority activity classes. Many also overlook the problem of class imbalance, which further degrades performance on underrepresented activities. The star operation, based on element-wise matrix multiplication, offers a lightweight alternative for capturing spatial and temporal dependencies without increasing computational cost. To jointly address efficiency and class imbalance in sensor-based HAR, we propose StarHAR, a low-latency and lightweight framework that integrates the star operation into the compact model architecture consisting of three components: a stem layer for low-level feature extraction, a sequence of star blocks for modeling hierarchical spatiotemporal relationships, and a classification head for activity prediction. To further enhance robustness under imbalanced data, we introduce a weighted loss function in the framework that promotes balanced learning across classes. Extensive experiments on five benchmark datasets and three hardware platforms show that StarHAR consistently outperforms state-of-the-art methods in both computational efficiency and classification accuracy, making it highly suitable for edge deployment.