We consider the problem of approximating the regression function \(f_\mu :\, \Omega \rightarrow Y\) from noisy \(\mu \) -distributed vector-valued data \((\omega _m,y_m)\in \Omega \times Y\) by an online learning algorithm using a reproducing kernel Hilbert space H (RKHS) as prior. In an online algorithm, i.i.d. samples become available one by one via a random process and are successively processed to build approximations to the regression function. Assuming that the regression function essentially belongs to H (soft learning scenario), we provide estimates for the expected squared error in the RKHS norm of the approximations \(f^{(m)}\in H\) obtained by a standard regularized online approximation algorithm. In particular, we show an order-optimal estimate \( \mathbb {E}(\Vert \epsilon ^{(m)}\Vert _H^2)\le C (m+1)^{-s/(2+s)},\qquad m=1,2,\ldots , \) where \(\epsilon ^{(m)}\) denotes the error term after m processed data, the parameter \(0<s\le 1\) expresses an additional smoothness assumption on the regression function, and the constant C depends on the variance of the input noise, the smoothness of the regression function, and other parameters of the algorithm. The proof, which is inspired by results on Schwarz iterative methods [12] in the noiseless case, uses only elementary Hilbert space techniques and minimal assumptions on the noise, the feature map that defines H and the associated covariance operator.
