With the proliferation of smart devices in personal and home environments, there is a growing need for cross-device interaction. However, distributed scenarios that cross device boundaries pose unique security and privacy challenges. While existing cross-device security mechanisms focus primarily on authentication, there is little research on fine-grained permission control. Permission models, which are critical security mechanisms for single devices, do not adequately support cross-device access control. To address this gap, we proposed and implemented a distributed role and attribute hybrid-based access control (DHBAC) model to enhance the security of cross-device access. DHBAC extends the single-device permission system to cross-device access control, providing fine-grained control based on users, devices, and applications. This approach effectively eliminates the over-authorization problem and supports the principle of least privilege. In addition, DHBAC can dynamically adjust and assign permissions based on specific scenarios and user requirements, improving the flexibility and adaptability of the system. To evaluate DHBAC, we deployed it on Harmony Operating System and tested it in several real-world, cross-device scenarios. Our evaluation shows that DHBAC effectively blocked malicious cross-device access and mitigated the associated security risks with acceptable system overhead.