mlpack

🔗 SparseCoding

The SparseCoding class implements sparse coding with dictionary learning using L1 regularization or elastic net (L1+L2) regularization. Sparse coding is a form of representation learning, and can be used to represent each point in a dataset as a sparse combination of atoms in the dictionary.

Simple usage example:

// Create a random dataset with 100 points in 40 dimensions, and then a random
// test dataset with 50 points.
arma::mat data(40, 100, arma::fill::randu);
arma::mat testData(40, 50, arma::fill::randu);

// Perform sparse coding with 20 atoms and an L1 penalty of 0.5.
mlpack::SparseCoding sc(20, 0.5);  // Step 1: create object.
double objective = sc.Train(data); // Step 2: learn dictionary.
arma::mat codes;
sc.Encode(testData, codes);        // Step 3: encode new data.

// Print some information about the test encoding.
std::cout << "Average density of encoded test data: "
    << 100.0 * arma::mean(arma::sum(codes != 0)) / codes.n_rows << "\%."
    << std::endl;

More examples...

See also:

🔗 Constructors

Constructor Parameters:

name type description default
data arma::mat Column-major training matrix. (N/A)
atoms size_t Number of atoms in dictionary. (N/A)
lambda1 double L1 regularization penalty. Used in both Train() and Encode() steps. 0.0
lambda2 double L2 regularization penalty (for elastic net regularization). Used in both Train() and Encode() steps. 0.0
maxIter size_t Maximum number of iterations for dictionary learning. 0 means no limit. 0
objTol double Objective function tolerance for terminating dictionary learning. 0.01
newtonTol double Tolerance for the Newton’s method dictionary optimization step. 1e-6

As an alternative to passing atoms, lambda1, lambda2, maxIter, objTol, or newtonTol, these can be set with a standalone method. The following functions can be used before calling Train():

Caveats:

🔗 Training

If training the dictionary is not done as part of the constructor call, it can be done with one of the following versions of the Train() member function:

🔗 Encoding

Once a SparseCoding model has a trained dictionary, the Encode() member function can be used to encode new data points.

After encoding, the original data can be recovered (approximately) as sc.Dictionary() * data.

🔗 Other Functionality

🔗 Simple Examples

See also the simple usage example for a trivial usage of the SparseCoding class.

Train a sparse coding model on the cloud dataset and print the reconstruction error.

// See https://datasets.mlpack.org/cloud.csv.
arma::mat dataset;
mlpack::data::Load("cloud.csv", dataset, true);

mlpack::SparseCoding sc;
sc.Atoms() = 50;
sc.Lambda1() = 0.1;
sc.Lambda2() = 0.001;
sc.MaxIterations() = 25;
sc.Train(dataset);

// Encode the training dataset.
arma::mat codes;
sc.Encode(dataset, codes);

std::cout << "Input matrix size: " << dataset.n_rows << " x " << dataset.n_cols
    << "." << std::endl;
std::cout << "Codes matrix size: " << codes.n_rows << " x " << codes.n_cols
    << "." << std::endl;

// Reconstruct the original matrix.
arma::mat recon = sc.Dictionary() * codes;
double error = std::sqrt(arma::norm(dataset - recon, "fro") / dataset.n_elem);
std::cout << "RMSE of reconstructed matrix: " << error << "." << std::endl;

Train a sparse coding model on the iris dataset and save the model to disk.

// See https://datasets.mlpack.org/iris.train.csv.
arma::mat dataset;
mlpack::data::Load("iris.train.csv", dataset, true);

// Train the model in the constructor.
mlpack::SparseCoding sc(dataset, 10 /* atoms */, 0.1 /* L1 penalty */);

// Save the model to disk.
mlpack::data::Save("sc.bin", "sc", sc);

Load a sparse coding model from disk and encode some new points from the iris dataset.

// Load model from disk.
mlpack::SparseCoding sc;
mlpack::data::Load("sc.bin", "sc", sc);

// See https://datasets.mlpack.org/iris.test.csv.
arma::mat dataset;
mlpack::data::Load("iris.test.csv", dataset, true);

// Encode the test points.
arma::mat codes;
sc.Encode(dataset, codes);

// Compute the sparse coding objective on the test points.
const double obj = sc.Objective(dataset, codes);
std::cout << "Sparse coding objective on test set: " << obj << "." << std::endl;

Train a sparse coding model on the satellite dataset, trying several different dictionary sizes and checking the objective value on a held-out test dataset.

// See https://datasets.mlpack.org/satellite.train.csv.
arma::mat trainData;
mlpack::data::Load("satellite.train.csv", trainData, true);
// See https://datasets.mlpack.org/satellite.test.csv.
arma::mat testData;
mlpack::data::Load("satellite.test.csv", testData, true);

for (size_t atoms = 20; atoms < 100; atoms += 10)
{
  mlpack::SparseCoding sc(atoms);
  sc.Lambda1() = 0.1;
  sc.MaxIterations() = 20; // Keep iterations low so this runs relatively fast.

  const double trainObj = sc.Train(trainData);

  // Compute the objective on the test set.
  arma::mat codes;
  sc.Encode(testData, codes);
  const double testObj = sc.Objective(testData, codes);

  std::cout << "Atoms: " << std::setfill(' ') << std::setw(3) << atoms << "; ";
  std::cout << "training set objective: " << std::setw(6) << trainObj << "; ";
  std::cout << "test set objective: " << std::setw(6) << testObj << "."
      << std::endl;
}

🔗 Advanced Functionality: Template Parameters

The SparseCoding class has one class template parameter that can be used for custom behavior. The full signature of the class is:

SparseCoding<MatType>

In addition, the constructors and Train() functions have a template parameter DictionaryInitializer that can be used for custom behavior.

MatType: Different Element Types

MatType specifies the type of matrix used for training data and internal representation of the dictionary. Any matrix type that implements the Armadillo API can be used. The example below trains a sparse coding model on 32-bit floating point data.

// See https://datasets.mlpack.org/cloud.csv.
arma::fmat dataset;
mlpack::data::Load("cloud.csv", dataset, true);

mlpack::SparseCoding<arma::fmat> sc;
sc.Atoms() = 30;
sc.Lambda1() = 0.15;
sc.Lambda2() = 0.001;
sc.MaxIterations() = 100;
// Note: a looser tolerance is often required when using floats instead of
// doubles.
sc.ObjTolerance() = 0.01;
sc.Train(dataset);

// Encode the training dataset.
arma::fmat codes;
sc.Encode(dataset, codes);

std::cout << "Input matrix size: " << dataset.n_rows << " x " << dataset.n_cols
    << "." << std::endl;
std::cout << "Codes matrix size: " << codes.n_rows << " x " << codes.n_cols
    << "." << std::endl;

// Reconstruct the original matrix.
arma::fmat recon = sc.Dictionary() * codes;
double error = std::sqrt(arma::norm(dataset - recon, "fro") / dataset.n_elem);
std::cout << "RMSE of reconstructed matrix: " << error << "." << std::endl;

DictionaryInitializer: Different Dictionary Initialization Strategies

The DictionaryInitializer template class specifies the strategy to be used to initialize the dictionary when Train() is called.

Note: none of the classes above have any members, and as such it is not necessary to use the constructor or Train() variants that take an initialized initializer object. That would only be necessary for a custom DictionaryInitializer class that stored internal members.

The example below uses NothingInitializer to set a specific initial dictionary.

// See https://datasets.mlpack.org/satellite.train.csv.
arma::mat trainData;
mlpack::data::Load("satellite.train.csv", trainData, true);

const size_t atoms = 25;
const double lambda1 = 0.1;
const double lambda2 = 0.0;
const size_t maxIterations = 50;

// Use a uniform random matrix as the initial dictionary.
arma::mat initialDictionary(trainData.n_rows, atoms, arma::fill::randu);

mlpack::SparseCoding sc(atoms, lambda1, lambda2, maxIterations);
sc.Dictionary() = initialDictionary;

const double obj = sc.Train<mlpack::NothingInitializer>(trainData);
std::cout << "Training set objective: " << obj << "." << std::endl;

// You can use this as a starting point for implementation.
class CustomDictionaryInitializer
{
 public:
  // Initialize the dictionary to have the given number of atoms, given the
  // dataset.  MatType will be the matrix type used by the sparse coding model
  // (e.g. `arma::mat`, `arma::fmat`, etc.).
  template<typename MatType>
  void Initialize(const MatType& data,
                  const size_t atoms,
                  MatType& dictionary);
};