With the development of intelligent devices such as drones and autonomous vehicles, high-precision navigation and positioning technology plays an important role in perception. However, the fixed integrated navigation architecture can no longer meet the high-precision navigation and positioning requirements in scenarios with varying navigation information sources. Therefore, this paper designs the multi-source sensors integrated navigation system framework that meets the ‘plug-and-play’ function. Firstly, building a high-precision inertial preintegration model. Then, establishing a factor node library model of multi-source sensors measurements built upon factor graph, and optimal system state is solved at a fixed frequency. Finally, considering the sensor fault, the residual chi-square method is used for detection, and the factor graph structure is adjusted according to detection results. Simulation results show that the method can combine information from various asynchronous navigation sensors successfully.