The Lure of Curvature
摘要
Some generic modelling- and interpretation- problems in multivariate calibration of multichannel instruments (chemometric “machine learning”) are addressed wrt extrapolation, interpolation and interpretation. Multi-wavelength high-speed diffuse spectrophotometry in e.g. the Near InfraRed (NIR) wavelength range are information rich but require “mathematical cleanup”: Whether they are measured as transmittance, interactance or reflectance, such high-speed real-world data are affected by several different types of variation. These combine to offer certain data modelling challenges, like multicollinearity, mixed additive-and-multiplicative effects and response curvature. Through a progression of linear and bilinear preprocessing and calibration steps, multivariate data modelling is shown to pick up and correct for these challenges, with a strong focus on statistical validation wrt overfitting and on linear extrapolation power. A particular focus is on how to handle curvature, implicitly and explicitly. The lure of curvature is that mathematically useful, but causally meaningless linear modelling effects of nonlinear curvature may be interpreted as “new and interesting spectral details”. This is illustrated with high-precision NIR transmittance spectra of powder mixtures, from an experiment especially designed to reveal the “dirty” effects of light absorption and scattering. A simulation example demonstrates the lure of curvature explicitly. Finally, a successful linear approximation of nonlinear curvature is illustrated conceptually, by the approximation of a curved (nonlinear) 3D banana by a flat (linear) 2D boomerang.