The development of a computational framework that can infer large-scale brain-wide effective connectivity (EC) based on resting-state functional MRI (rs-fMRI) represents a grand challenge to computational neuroimaging. Towards the goal of estimating full-scale, whole-brain EC, we developed a new computational framework termed Large-scale nEural Model Inversion (LEMI) by utilizing a linear neural mass model with an efficient Kalman-filter based gradient descent algorithm. Key advantages of LEMI include fast estimation of both intra-regional and inter-regional connection strengths for large-scale networks, allowing exploration of both intrinsic and external mechanisms in neuroscience problems. Using ground-truth simulations, we demonstrated that LEMI can accurately and efficiently recover model parameters in a large network (100 regions) within 90 min. We then applied the LEMI model to an empirical rs-fMRI dataset from the ADNI database and identified widespread reduced excitation-inhibition (E-I) ratio in patients with Alzheimer’s disease (AD). Overall, LEMI provides an efficient and accurate computational framework to estimate large-scale EC and whole-brain E-I balance based on non-invasive neuroimaging data.