Suppose we are interested in studying and learning about a certain phenomenon that can be schematically represented as a system that takes an “input” X and generates or associates with X a “response” Y . Let D = {(X1, Y1), . . . , (Xn, Yn)} be the data, a collection of inputs and the corresponding responses, obtained by experiment or observation of the phenomenon. Regression analysis is the study of dependence between X and Y . The primary goal of regression analysis is to construct a stochastic model for predicting response Y from a new input X based on the observed data D. Regression analysis originated in the early XIX century in the works of Adrien-Marie Legendre and Carl Gauss, who used astronomical observations for studying the orbits of planets and comets around the Sun. The theory of regression and its methods were further developed, and applications were extended to other domains by Francis Galton, Karl Pearson, and Ronald Fisher. Regression analysis is still an active area of modern research and its methods play a central role in machine learning, information and data sciences. In this chapter, we will study the simplest version of the model, namely, the simple linear regression model.