mlpack

🔗 AdaBoost

The AdaBoost class implements the ‘adaptive boosting’ classifier AdaBoost.MH. This classifier is an ensemble of weak learners. The AdaBoost class offers control over the weak learners and other behavior via template parameters. By default, the Perceptron class is used as a weak learner.

AdaBoost is useful for classifying points with discrete labels (i.e. 0, 1, 2).

Simple usage example:

Train an AdaBoost model on random data and predict labels on a random test set.

// Train an AdaBoost model on random data and predict labels on test data:

// All data and labels are uniform random; 10 dimensional data, 5 classes.
// Replace with a data::Load() call or similar for a real application.
arma::mat dataset(10, 1000, arma::fill::randu); // 1000 points.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));
arma::mat testDataset(10, 500, arma::fill::randu); // 500 test points.

mlpack::AdaBoost ab;                   // Step 1: create model.
ab.Train(dataset, labels, 5);          // Step 2: train model.
arma::Row<size_t> predictions;
ab.Classify(testDataset, predictions); // Step 3: classify points.

// Print some information about the test predictions.
std::cout << arma::accu(predictions == 3) << " test points classified as class "
    << "3." << std::endl;

More examples...

See also:

🔗 Constructors

Constructor Parameters:

name type description default
data arma::mat Column-major training matrix. (N/A)
labels arma::Row<size_t> Training labels, between 0 and numClasses - 1 (inclusive). Should have length data.n_cols. (N/A)
numClasses size_t Number of classes in the dataset. (N/A)
weakLearner Perceptron An initialized weak learner whose hyperparameters will be used as settings for weak learners during training. (N/A)
maxIterations size_t Maximum number of iterations of AdaBoost.MH to use. This is the maximum number of weak learners to train. (0 means no limit, and weak learners will be trained until the tolerance is met.) 100
tolerance double When the weighted residual (r_t) of the model goes below tolerance, training will terminate and no more weak learners will be added. 1e-6

As an alternative to passing hyperparameters, each hyperparameter can be set with a standalone method. The following functions can be used before calling Train() to set hyperparameters:

Note: different types of weak learners can be used than Perceptron, by changing the WeakLearnerType template parameter.

🔗 Training

If training is not done as part of the constructor call, it can be done with one of the versions of the Train() member function. For an instance of AdaBoost named ab, the following functions for training are available:

Types of each argument are the same as in the table for constructors above.

Notes:

🔗 Classification

Once an AdaBoost model is trained, the Classify() member function can be used to make class predictions for new data.

Classification Parameters:

usage name type description
single-point point arma::vec Single point for classification.
single-point prediction size_t& size_t to store class prediction into.
single-point probabilitiesVec arma::vec& arma::vec& to store class probabilities into.
       
multi-point data arma::mat Set of column-major points for classification.
multi-point predictions arma::Row<size_t>& Vector of size_ts to store class prediction into; will be set to length data.n_cols.
multi-point probabilities arma::mat& Matrix to store class probabilities into (number of rows will be equal to number of classes; number of columns will be equal to data.n_cols).

🔗 Other Functionality

For complete functionality, the source code can be consulted. Each method is fully documented.

🔗 Simple Examples

See also the simple usage example for a trivial usage of the AdaBoost class.

Train an AdaBoost model using the hyperparameters from an existing weak learner.

// See https://datasets.mlpack.org/iris.csv.
arma::mat dataset;
mlpack::data::Load("iris.csv", dataset, true);
// See https://datasets.mlpack.org/iris.labels.csv.
arma::Row<size_t> labels;
mlpack::data::Load("iris.labels.csv", labels, true);

mlpack::AdaBoost ab;
// Train with a custom number of perceptron iterations, and custom AdaBoost
// parameters.
ab.Train(dataset, labels, 3, 75 /* maximum number of weak learners */,
                             1e-6 /* tolerance for AdaBoost convergence */,
                             100 /* maximum number of perceptron iterations */);

// Now predict the label of a point and the probabilities of each class.
size_t prediction;
arma::rowvec probabilities;
ab.Classify(dataset.col(10), prediction, probabilities);

std::cout << "Point 11 is predicted to have class " << prediction << "."
    << std::endl;
std::cout << "Probabilities of each class: " << probabilities;

Before training an AdaBoost model, set hyperparameters individually. Save the trained model to disk.

// See https://datasets.mlpack.org/iris.csv.
arma::mat dataset;
mlpack::data::Load("iris.csv", dataset, true);
// See https://datasets.mlpack.org/iris.labels.csv.
arma::Row<size_t> labels;
mlpack::data::Load("iris.labels.csv", labels, true);

mlpack::AdaBoost ab;
ab.MaxIterations() = 50; // Use at most 50 weak learners.
ab.Tolerance() = 1e-4; // Set a custom tolerance for convergence.

// Now train, using the hyperparameters specified above.
ab.Train(dataset, labels, 3);

// Save the model to `adaboost_model.bin`.
mlpack::data::Save("adaboost_model.bin", "adaboost_model", ab, true);

Load an AdaBoost model and print some information about it.

// Load a saved model named "adaboost_model" from `adaboost_model.bin`.
mlpack::AdaBoost ab;
mlpack::data::Load("adaboost_model.bin", "adaboost_model", ab, true);

std::cout << "Details about the model in `adaboost_model.bin`:" << std::endl;
std::cout << "  - Trained on " << ab.NumClasses() << " classes." << std::endl;
std::cout << "  - Tolerance used for training: " << ab.Tolerance() << "."
    << std::endl;
std::cout << "  - Number of perceptron weak learners in model"
    << ab.WeakLearners() << "." << std::endl;

// Print some details about the first weak learner, if available.  The weak
// learner type is `Perceptron<>`.
if (ab.WeakLearners() > 0)
{
  std::cout << "  - Weight of first perceptron weak learner: " << ab.Alpha(0)
      << "." << std::endl;
  std::cout << "  - Biases of first perceptron learner: "
      << ab.WeakLearner(0).Biases().t();
}

See also the following fully-working examples:

🔗 Advanced Functionality: Template Parameters

The AdaBoost class has two template parameters that can be used for custom behavior. The full signature of the class is:

AdaBoost<WeakLearnerType, MatType>

WeakLearnerType

// You can use this as a starting point for implementation.
class CustomWeakLearner
{
 public:
  // Train the model with the given hyperparameters.
  //
  //  * `MatType` will be an Armadillo-like matrix type (typically `arma::mat`);
  //    this is the same type as the `MatType` template parameter for the
  //    `AdaBoost` class.
  //
  //  * `data` and `labels` are the same dataset passed to the `Train()` method
  //    of `AdaBoost`.
  //
  //  * `weights` contains instance weights for each point (column) of `data`.
  //
  // Note: there is no restriction on the number or types of hyperparameters
  // that can be used, but, they do need default arguments.  The example here
  // includes two.
  template<typename MatType>
  void Train(const MatType& data,
             const arma::Row<size_t>& labels,
             const size_t numClasses,
             const arma::rowvec& weights,
             const size_t hyperparameterA = 10,
             const double hyperparameterB = 0.1);

  // Classify the given point.  `VecType` will be an Armadillo-like type that is
  // a vector that represents a single point.
  template<typename VecType>
  size_t Classify(const VecType& point);

  // Classify the given points in `data`, storing the predicted classifications
  // in `predictions`.
  template<typename MatType>
  void Classify(const MatType& data, arma::Row<size_t>& predictions);
};

MatType

🔗 Advanced Functionality Examples

Train an AdaBoost model using decision stumps as the weak learner (use a different WeakLearnerType).

// 1000 random points in 10 dimensions.
arma::mat dataset(10, 1000, arma::fill::randu);
// Random labels for each point, totaling 5 classes.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));

// Train in the constructor.
// Note that we specify decision stumps as the weak learner type, and pass
// hyperparameters for the decision stump (these could be omitted).  See the
// DecisionTree documentation for more details on the ID3DecisionStump-specific
// hyperparameters.
mlpack::AdaBoost<mlpack::ID3DecisionStump> ab(dataset, labels, 5,
    25 /* maximum number of decision stumps */,
    1e-6 /* tolerance for convergence of AdaBoost */,
    /** Hyperparameters specific to ID3DecisionStump: **/
    10 /* minimum number of points in each leaf of the decision stump */,
    1e-5 /* minimum gain for splitting the root node of the decision stump */);

// Create test data (500 points).
arma::mat testDataset(10, 500, arma::fill::randu);
arma::Row<size_t> predictions;
ab.Classify(testDataset, predictions);
// Now `predictions` holds predictions for the test dataset.

// Print some information about the test predictions.
std::cout << arma::accu(predictions == 3) << " test points classified as class "
    << "3." << std::endl;

Train an AdaBoost model on 32-bit floating-point precision data (use a different MatType).

// 1000 random points in 10 dimensions, using 32-bit precision (float).
arma::fmat dataset(10, 1000, arma::fill::randu);
// Random labels for each point, totaling 5 classes.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));

// Train in the constructor, using floating-point data.
// The weak learner type is now a floating-point Perceptron.
using PerceptronType = mlpack::Perceptron<mlpack::SimpleWeightUpdate,
                                          mlpack::ZeroInitialization,
                                          arma::fmat>;
mlpack::AdaBoost<PerceptronType, arma::fmat> ab(dataset, labels, 5);

// Create test data (500 points).
arma::fmat testDataset(10, 500, arma::fill::randu);
arma::Row<size_t> predictions;
ab.Classify(testDataset, predictions);
// Now `predictions` holds predictions for the test dataset.

// Print some information about the test predictions.
std::cout << arma::accu(predictions == 3) << " test points classified as class "
    << "3." << std::endl;