Human activity monitoring extends across numerous application scenarios, including elderly care and health wellness systems. Currently, wearable devices may affect the daily life of the human body. Camera-based activity monitoring may raise privacy concerns. Although the application of millimeter-wave radar in human activity monitoring has been extensively studied, the issues of point cloud confusion and trajectory continuity when multiple people are close to each other in multi-person scenarios have not been well resolved. To address these challenges, we propose an indoor multi-person activity monitoring system utilizing millimeter wave radar, which can solve the points cloud fusion problem and trajectory continuity problem when multiple people are close to each other in home scenes. In addition, we propose a highly accurate and practical dual-transformer network based on the Transformer architecture to fully extract the action features of sparse point clouds and solve the problem of inconsistent point numbers in each frame. Ultimately, we deploy the system and construct a human activity dataset for experimental validation. The trials demonstrate that our system attained an accuracy of 94.5% in single-person real-time scenarios and 89.09% in the more challenging multi-person real-time scenarios.