Gaussian Process Regression

Gaussian process regression is one of the most powerful regression methods that you could potentially use. Its characteristics are that it is Bayesian and non-parametric, i.e., it does not require you to specify a parametric model like generalized linear model. Instead, you start by quantifying your state of knowledge about the function you are trying to learn by stating how regular you expect it to be, e.g., how many continuous derivatives is it going to have, how fast you expect the function values to change when you change each one of the function inputs, how much you expect the function values to vary. Then you simply condition this prior knowledge on the observed data. Gaussian process regression is particularly useful when you do not have a lot of input-output observations. We end the chapter with an application of Gaussian process regression to optimization of very expensive to evaluate objective functions.