BayesianLinearRegression Class Reference

A Bayesian approach to the maximum likelihood estimation of the parameters $ \omega $ of the linear regression model. More...

Public Member Functions

 BayesianLinearRegression (const bool centerData=true, const bool scaleData=false, const size_t nIterMax=50, const double tol=1e-4)
 Set the parameters of Bayesian Ridge regression object. More...

 
double Alpha () const
 Get the precision (or inverse variance) of the gaussian prior. More...

 
double Beta () const
 Get the precision (or inverse variance) beta of the model. More...

 
const arma::colvec & DataOffset () const
 Get the mean vector computed on the features over the training points. More...

 
const arma::colvec & DataScale () const
 Get the vector of standard deviations computed on the features over the training points. More...

 
const arma::colvec & Omega () const
 Get the solution vector. More...

 
void Predict (const arma::mat &points, arma::rowvec &predictions) const
 Predict $y_{i}$ for each data point in the given data matrix using the currently-trained Bayesian Ridge model. More...

 
void Predict (const arma::mat &points, arma::rowvec &predictions, arma::rowvec &std) const
 Predict $y_{i}$ and the standard deviation of the predictive posterior distribution for each data point in the given data matrix, using the currently-trained Bayesian Ridge estimator. More...

 
double ResponsesOffset () const
 Get the mean value of the train responses. More...

 
double RMSE (const arma::mat &data, const arma::rowvec &responses) const
 Compute the Root Mean Square Error between the predictions returned by the model and the true responses. More...

 
template
<
typename
Archive
>
void serialize (Archive &ar, const unsigned int)
 Serialize the BayesianLinearRegression model. More...

 
double Train (const arma::mat &data, const arma::rowvec &responses)
 Run BayesianLinearRegression. More...

 
double Variance () const
 Get the estimated variance. More...

 

Detailed Description

A Bayesian approach to the maximum likelihood estimation of the parameters $ \omega $ of the linear regression model.

The Complexity is governed by the addition of a gaussian isotropic prior of precision $ \alpha $ over $ \omega $:

\[ p(\omega|\alpha) = \mathcal{N}(\omega|0, \alpha^{-1}I) \]

The optimization procedure calculates the posterior distribution of $ \omega $ knowing the data by maximizing an approximation of the log marginal likelihood derived from a type II maximum likelihood approximation. The determination of $ alpha $ and of the noise precision $ beta $ is part of the optimization process, leading to an automatic determination of w. The model being entirely based on probabilty distributions, uncertainties are available and easly computed for both the parameters and the predictions.

The advantage over linear regression and ridge regression is that the regularization is determined from all the training data alone without any require to an hold out method.

The code below is an implementation of the maximization of the evidence function described in the section 3.5.2 of the C.Bishop book, Pattern Recognition and Machine Learning.

@article{MacKay91bayesianinterpolation,
author = {David J.C. MacKay},
title = {Bayesian Interpolation},
journal = {NEURAL COMPUTATION},
year = {1991},
volume = {4},
pages = {415--447}
}
@book{Bishop:2006:PRM:1162264,
author = {Bishop, Christopher M.},
title = {Pattern Recognition and Machine Learning (Information Science
and Statistics)},
chapter = {3}
year = {2006},
isbn = {0387310738},
publisher = {Springer-Verlag},
address = {Berlin, Heidelberg},
}

Example of use:

arma::mat xTrain; // Train data matrix. Column-major.
arma::rowvec yTrain; // Train target values.
// Train the model. Regularization strength is optimally tunned with the
// training data alone by applying the Train method.
BayesianLinearRegression estimator(); // Instanciate the estimator with default option.
estimator.Train(xTrain, yTrain);
// Prediction on test points.
arma::mat xTest; // Test data matrix. Column-major.
arma::rowvec predictions;
estimator.Predict(xTest, prediction);
arma::rowvec yTest; // Test target values.
estimator.RMSE(xTest, yTest); // Evaluate using the RMSE score.
// Compute the standard deviations of the predictions.
arma::rowvec stds;
estimator.Predict(xTest, responses, stds)

Definition at line 98 of file bayesian_linear_regression.hpp.

Constructor & Destructor Documentation

◆ BayesianLinearRegression()

BayesianLinearRegression ( const bool  centerData = true,
const bool  scaleData = false,
const size_t  nIterMax = 50,
const double  tol = 1e-4 
)

Set the parameters of Bayesian Ridge regression object.

The regularization parameter is automatically set to its optimal value by maximization of the marginal likelihood.

Parameters
centerDataWhether or not center the data according to the examples.
scaleDataWhether or not scale the data according to the standard deviation of each feature.
nIterMaxMaximum number of iterations for convergency.
tolLevel from which the solution is considered sufficientlly stable.

Member Function Documentation

◆ Alpha()

double Alpha ( ) const
inline

Get the precision (or inverse variance) of the gaussian prior.

Train() must be called before.

Returns
$ \alpha $

Definition at line 178 of file bayesian_linear_regression.hpp.

◆ Beta()

double Beta ( ) const
inline

Get the precision (or inverse variance) beta of the model.

Train() must be called before.

Returns
$ \beta $

Definition at line 186 of file bayesian_linear_regression.hpp.

Referenced by BayesianLinearRegression::Variance().

◆ DataOffset()

const arma::colvec& DataOffset ( ) const
inline

Get the mean vector computed on the features over the training points.

Returns
responsesOffset

Definition at line 200 of file bayesian_linear_regression.hpp.

◆ DataScale()

const arma::colvec& DataScale ( ) const
inline

Get the vector of standard deviations computed on the features over the training points.

Returns
dataOffset

Definition at line 208 of file bayesian_linear_regression.hpp.

◆ Omega()

const arma::colvec& Omega ( ) const
inline

Get the solution vector.

Returns
omega Solution vector.

Definition at line 170 of file bayesian_linear_regression.hpp.

◆ Predict() [1/2]

void Predict ( const arma::mat &  points,
arma::rowvec &  predictions 
) const

Predict $y_{i}$ for each data point in the given data matrix using the currently-trained Bayesian Ridge model.

Parameters
pointsThe data points to apply the model.
predictionsy, Contains the predicted values on completion.
Returns
Root mean squared error computed on the train set.

◆ Predict() [2/2]

void Predict ( const arma::mat &  points,
arma::rowvec &  predictions,
arma::rowvec &  std 
) const

Predict $y_{i}$ and the standard deviation of the predictive posterior distribution for each data point in the given data matrix, using the currently-trained Bayesian Ridge estimator.

Parameters
pointsThe data point to apply the model.
predictionsVector which will contain calculated values on completion.
stdStandard deviations of the predictions.

◆ ResponsesOffset()

double ResponsesOffset ( ) const
inline

Get the mean value of the train responses.

Returns
responsesOffset

Definition at line 215 of file bayesian_linear_regression.hpp.

References BayesianLinearRegression::serialize().

◆ RMSE()

double RMSE ( const arma::mat &  data,
const arma::rowvec &  responses 
) const

Compute the Root Mean Square Error between the predictions returned by the model and the true responses.

Parameters
dataData points to predict
responsesA vector of targets.
Returns
Root mean squared error.

◆ serialize()

void serialize ( Archive &  ar,
const unsigned  int 
)

◆ Train()

double Train ( const arma::mat &  data,
const arma::rowvec &  responses 
)

Run BayesianLinearRegression.

The input matrix (like all mlpack matrices) should be column-major – each column is an observation and each row is a dimension.

Parameters
dataColumn-major input data, dim(P, N).
responsesA vector of targets, dim(N).
Returns
Root mean squared error.

◆ Variance()

double Variance ( ) const
inline

Get the estimated variance.

Train() must be called before.

Returns
1.0 / $ \beta $

Definition at line 193 of file bayesian_linear_regression.hpp.

References BayesianLinearRegression::Beta().


The documentation for this class was generated from the following file: