Accurate battery mathematical models are essential to estimate the lithium-ion battery state of charge (SOC). The conventional equivalent circuit model, however, does not describe the actual electrochemical nonlinear dynamic response of a lithium-ion battery. In this work, a novel nonlinear equivalent circuit model (NLECM) is established, which is based on pulse-multisine signal for parameter estimation. Based on the established NLECM model, a window variational adaptive extended Kalman filter (WVAEKF) is first applied for SOC estimation. The designed WVAEKF can identify variations in the error innovation sequence distribution and modify the window’s length. The experimental results demonstrate that the WVAEKF algorithm’s estimation of SOC has an error limit of 1%. Compared with the traditional AEKF algorithm with a fixed window length of 6, the RMSE of SOC estimation based on WVAEKF algorithm is reduced by 23%.