In order to analyze the effect of nano-SiO2 and its surface silane coupling agent modification on the performance of epoxy/anhydride system from the microscopic system, this paper adopted the molecular dynamics method to establish the models of pure epoxy resin, SiO2/epoxy resin and modified SiO2/epoxy resin composites, and investigated the rationality of the models of three kinds of systems, the mechanical properties, the glass transition temperature and the free volume. The results show that: the density and energy of the model tend to be stabilized in the process of relaxation, which proves the reasonableness of the three models; the mechanical properties and glass transition temperature of SiO2 and modified SiO2 were greatly enhanced compared to pure epoxy resin. The model with KH550 grafting showed the most notable improvement. The addition of nano-SiO2 altered the microstructure of the epoxy resin system, resulting in reduced free volume, with modified SiO2 having a more pronounced effect.