mlpack

🔗 LinearRegression

The LinearRegression class implements a standard L2-regularized linear regression model for numerical data, trained by direct decomposition of the training data. The class offers configurable functionality and template parameters to control the data type used for storing the model.

Simple usage example:

// Train a linear regression model on random numeric data and make predictions.

// All data and responses are uniform random; this uses 10 dimensional data.
// Replace with a data::Load() call or similar for a real application.
arma::mat dataset(10, 1000, arma::fill::randu); // 1000 points.
arma::rowvec responses = arma::randn<arma::rowvec>(1000);
arma::mat testDataset(10, 500, arma::fill::randu); // 500 test points.

mlpack::LinearRegression lr;          // Step 1: create model.
lr.Train(dataset, responses);         // Step 2: train model.
arma::rowvec predictions;
lr.Predict(testDataset, predictions); // Step 3: use model to predict.

// Print some information about the test predictions.
std::cout << arma::accu(predictions > 0.7) << " test points predicted to have"
    << " responses greater than 0.7." << std::endl;
std::cout << arma::accu(predictions < 0) << " test points predicted to have "
    << "negative responses." << std::endl;

More examples...

See also:

🔗 Constructors

Constructor Parameters:

name type description default
data arma::mat Column-major training matrix. (N/A)
responses arma::rowvec Training responses (e.g. values to predict). Should have length data.n_cols. (N/A)
weights arma::rowvec Weights for each training point. Should have length data.n_cols. (N/A)
lambda double L2 regularization penalty parameter. 0.0
intercept bool Whether to fit an intercept term in the model. bool

As an alternative to passing lambda, it can be set with the standalone Lambda() method: lr.Lambda() = l; will set the value of lambda to l for the next time Train() is called.

Note: setting lambda too small may cause the model to overfit; however, setting it too large may cause the model to underfit. Automatic hyperparameter tuning can be used to find a good value of lambda instead of a manual setting.

🔗 Training

If training is not done as part of the constructor call, it can be done with the Train() function:

Types of each argument are the same as in the table for constructors above.

Notes:

🔗 Prediction

Once a LinearRegression model is trained, the Predict() member function can be used to make predictions for new data.

Prediction Parameters:

usage name type description
single-point point arma::vec Single point for prediction.
       
multi-point data arma::mat Set of column-major points for classification.
multi-point predictions arma::rowvec& Vector of doubles to store predictions into. Will be set to length data.n_cols.

🔗 Other Functionality

🔗 Simple Examples

See also the simple usage example for a trivial usage of the LinearRegression class.

Train a linear regression model in the constructor on weighted data, compute the objective function with ComputeError(), and save the model.

// See https://datasets.mlpack.org/admission_predict.csv.
arma::mat data;
mlpack::data::Load("admission_predict.csv", data, true);

// See https://datasets.mlpack.org/admission_predict.responses.csv.
arma::rowvec responses;
mlpack::data::Load("admission_predict.responses.csv", responses, true);

// Generate random instance weights for each point, in the range 0.5 to 1.5.
arma::rowvec weights(data.n_cols, arma::fill::randu);
weights += 0.5;

// Train a linear regression model, fitting an intercept term and using an L2
// regularization parameter of 0.3.
mlpack::LinearRegression lr(data, responses, weights, 0.3, true);

// Now compute the MSE on the training set.
std::cout << "MSE on the training set: " << lr.ComputeError(data, responses)
    << "." << std::endl;

// Finally, save the model with the name "lr".
mlpack::data::Save("lr_model.bin", "lr", lr, true);

Load a saved linear regression model and print some information about it, then make some predictions individually for random points.

mlpack::LinearRegression lr;

// Load the model named "lr" from "lr_model.bin".
mlpack::data::Load("lr_model.bin", "lr", lr, true);

// Print some information about the model.
const size_t dimensionality =
    (lr.Intercept() ? (lr.Parameters().n_elem - 1) : lr.Parameters().n_elem);

std::cout << "Information on the LinearRegression model in 'lr_model.bin':"
    << std::endl;
std::cout << " - Model has intercept: "
    << (lr.Intercept() ? std::string("yes") : std::string("no")) << "."
    << std::endl;
if (lr.Intercept())
{
  std::cout << " - Intercept weight: " << lr.Parameters()[0] << "."
      << std::endl;
}
std::cout << " - Model dimensionality: " << dimensionality << "." << std::endl;
std::cout << " - Lambda value: " << lr.Lambda() << "." << std::endl;
std::cout << std::endl;

// Now make a prediction for three random points.
for (size_t t = 0; t < 3; ++t)
{
  arma::vec randomPoint(dimensionality, arma::fill::randu);
  const double prediction = lr.Predict(randomPoint);

  std::cout << "Prediction for random point " << t << ": " << prediction << "."
      << std::endl;
}

See also the following fully-working examples:

🔗 Advanced Functionality: Different Element Types

The LinearRegression class has one template parameter that can be used to control the element type of the model. The full signature of the class is:

LinearRegression<ModelMatType>

ModelMatType specifies the type of matrix used for the internal representation of model parameters. Any matrix type that implements the Armadillo API can be used.

Note that the Train() and Predict() functions themselves are templatized and can allow any matrix type that has the same element type. So, for instance, a LinearRegression<arma::mat> can accept an arma::sp_mat for training.

The example below trains a linear regression model on sparse 32-bit floating point data, but uses a dense 32-bit floating point vector to store the model itself.

// Create random, sparse 100-dimensional data.
arma::sp_fmat dataset;
dataset.sprandu(100, 5000, 0.3);

// Generate noisy responses from random data.
arma::fvec trueWeights(100, arma::fill::randu);
arma::frowvec responses = trueWeights.t() * dataset +
    0.01 * arma::randu<arma::frowvec>(5000) /* noise term */;

mlpack::LinearRegression<arma::fmat> lr;
lr.Lambda() = 0.01;

lr.Train(dataset, responses);

// Compute the MSE on the training set and a random test set.
arma::sp_fmat testDataset;
testDataset.sprandu(100, 1000, 0.3);

arma::frowvec testResponses = trueWeights.t() * testDataset +
    0.01 * arma::randu<arma::frowvec>(1000) /* noise term */;

std::cout << "MSE on training set: "
    << lr.ComputeError(dataset, responses) << "." << std::endl;
std::cout << "MSE on test set:     "
    << lr.ComputeError(testDataset, testResponses) << "." << std::endl;

Note: dense objects should be used for ModelMatType, since in general an L2-regularized linear regression model will not be sparse.