<p>Mathematical models are routinely applied to interpret biological data, with common goals that include both prediction and parameter estimation. A challenge in mathematical biology, in particular, is that models are often complex and non-identifiable, while data are limited. Rectifying identifiability through simplification can seemingly yield more precise parameter estimates, albeit, as we explore in this perspective, at the potentially catastrophic cost of introducing model misspecification and poor accuracy. We demonstrate how uncertainty in model structure can be propagated through to uncertainty in parameter estimates using a semi-parametric Gaussian process approach that delineates parameters of interest from uncertainty in model terms. Specifically, we study generalised logistic growth with an unknown crowding function, and a spatially resolved process described by a partial differential equation with a time-dependent diffusivity parameter. Allowing for structural model uncertainty yields more robust and accurate parameter estimates, and a better quantification of remaining uncertainty. We conclude our perspective by discussing the connections between identifiability and model misspecification, and alternative approaches to dealing with model misspecification in mathematical biology.</p>