In software engineering, complex system architecture design faces problems such as frequent changes in requirements, difficulty in system integration, and poor maintainability. There is an urgent need for effective methodology to improve the flexibility and reliability of the design. This paper aims to explore the application of ER (Entity-Relationship) algorithm in complex system architecture design to optimize system modeling and data management. First, through demand analysis, the functional and data requirements of the system are collected to identify key users and usage scenarios. Then, the core entities and their attributes in the system are identified, and the relationships between entities are established. Finally, the ER diagram tool is used to visualize the entities and their relationships to form a clear model to facilitate communication and confirmation with the development team. By simulating system usage scenarios, it was verified that the number of entity relationships in the complexity of system design was reduced by 24% after the application of the ER algorithm. The ER algorithm provides an effective modeling tool for complex system architecture design, which can significantly improve design quality and efficiency when responding to dynamic requirements.