mlpack

Kernels

mlpack includes a number of Mercer kernels for its kernel-based techniques. These all implement the same API, providing one Evaluate() method. mlpack provides a number of supported kernels:

These kernels can then be used in a number of machine learning algorithms that mlpack provides:

🔗 GaussianKernel

The GaussianKernel class implements the standard Gaussian kernel (also called the radial basis function kernel or RBF kernel).

The Gaussian kernel is defined as: k(x1, x2) = exp(-|| x1 - x2 ||^2 / (2 * bw^2)) where bw is the bandwidth parameter of the kernel.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Other utilities

🔗 Example usage

// Create a Gaussian kernel with default bandwidth.
mlpack::GaussianKernel g;

// Create a Gaussian kernel with bandwidth 5.0.
mlpack::GaussianKernel g2(5.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = g.Evaluate(x1, x2);
const double k2 = g2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (bw=1.0), " << k2 << " (bw=5.0)."
    << std::endl;

// Evaluate the kernel value when the distance between two points is already
// computed.
const double distance = 1.5;
const double k3 = g.Evaluate(distance);

// Change the bandwidth of the kernel to 2.5.
g.Bandwidth(2.5);
const double k4 = g.Evaluate(x1, x2);
std::cout << "Kernel value with bw=2.5: " << k4 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = g.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 1.5");
arma::fvec fx2("1.5 1.0 0.5");
const double k5 = g.Evaluate(fx1, fx2);
const double k6 = g2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k5
    << " (bw=2.5), " << k6 << " (bw=5.0)." << std::endl;

🔗 CauchyKernel

The CauchyKernel class implements the Cauchy kernel, a kernel function with a longer tail than the Gaussian kernel, defined as: k(x1, x2) = 1 / (1 + (|| x1 - x2 ||^2 / bw^2)) where bw is the bandwidth parameter of the kernel.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Example usage

// Create a Cauchy kernel with default bandwidth.
mlpack::CauchyKernel c;

// Create a Cauchy kernel with bandwidth 5.0.
mlpack::CauchyKernel c2(5.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = c.Evaluate(x1, x2);
const double k2 = c2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (bw=1.0), " << k2 << " (bw=5.0)."
    << std::endl;

// Change the bandwidth of the kernel to 2.5.
c.Bandwidth(2.5);
const double k3 = c.Evaluate(x1, x2);
std::cout << "Kernel value with bw=2.5: " << k3 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = c.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 1.5");
arma::fvec fx2("1.5 1.0 0.5");
const double k4 = c.Evaluate(fx1, fx2);
const double k5 = c2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k4
    << " (bw=2.5), " << k5 << " (bw=5.0)." << std::endl;

🔗 CosineSimilarity

The CosineSimilarity class implements the dot-product cosine similarity, defined as: k(x1, x2) = (x1^T x2) / (|| x1 || * || x2 ||). The value of the kernel is limited to the range [-1, 1]. The cosine similarity is often used in text mining tasks.

🔗 Constructor

Note: because the CosineSimilarity kernel has no parameters, it is not necessary to create an object and the Evaluate() function (below) can be called statically.

🔗 Kernel evaluation

🔗 Example usage

// Create a cosine similarity kernel.
mlpack::CosineSimilarity c;

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = c.Evaluate(x1, x2);
const double k2 = c.Evaluate(x1, x1);
const double k3 = c.Evaluate(x2, x2);
std::cout << "Cosine similarity values:" << std::endl;
std::cout << "  - k(x1, x2): " << k1 << "." << std::endl;
std::cout << "  - k(x1, x1): " << k2 << "." << std::endl;
std::cout << "  - k(x2, x2): " << k3 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix,
// using the static Evaluate() function.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = mlpack::CosineSimilarity::Evaluate(x1, r.col(i));
std::cout << "Average cosine similarity for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the cosine similarity between two sparse 32-bit floating point
// vectors.
arma::sp_fvec x3, x4;
x3.sprandu(100, 1, 0.2);
x4.sprandu(100, 1, 0.2);
const double k4 = mlpack::CosineSimilarity::Evaluate(x3, x4);
std::cout << "Cosine similarity between two random sparse 32-bit floating "
    << "point vectors: " << k4 << "." << std::endl;

🔗 EpanechnikovKernel

The EpanechnikovKernel implements the parabolic or Epanechnikov kernel, defined as the following function: k(x1, x2) = max(0, (3 / 4) * (1 - (|| x1 - x2 ||_2 / bw)^2)), where bw is the bandwidth parameter of the kernel.

The kernel takes the value 0 when || x1 - x2 ||_2 (the Euclidean distance between x1 and x2) is greater than or equal to bw.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Other utilities

🔗 Example usage

// Create an Epanechnikov kernel with default bandwidth.
mlpack::EpanechnikovKernel e;

// Create an Epanechnikov kernel with bandwidth 5.0.
mlpack::EpanechnikovKernel e2(5.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = e.Evaluate(x1, x2);
const double k2 = e2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (bw=1.0), " << k2 << " (bw=5.0)."
    << std::endl;

// Evaluate the kernel value when the distance between two points is already
// computed.
const double distance = 1.5;
const double k3 = e.Evaluate(distance);

// Change the bandwidth of the kernel to 2.5.
e.Bandwidth(2.5);
const double k4 = e.Evaluate(x1, x2);
std::cout << "Kernel value with bw=2.5: " << k4 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = e.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 1.5");
arma::fvec fx2("1.5 1.0 0.5");
const double k5 = e.Evaluate(fx1, fx2);
const double k6 = e2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k5
    << " (bw=2.5), " << k6 << " (bw=5.0)." << std::endl;

🔗 HyperbolicTangentKernel

The HyperbolicTangentKernel implements the hyperbolic tangent kernel, which is defined by the following equation: f(x1, x2) = tanh(s * (x1^T x2) + t) where s is the scale parameter and t is the offset parameter.

The hyperbolic tangent kernel is not a positive definite Mercer kernel and thus does not satisfy the theoretical requirements of many kernel methods. See this discussion for more details. In practice, for many kernel methods, it may still provide compelling results despite this theoretical limitation.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Example usage

// Create a hyperbolic tangent kernel with default scale and offset.
mlpack::HyperbolicTangentKernel h;

// Create a hyperbolic tangent kernel with scale 2.0 and offset 1.0.
mlpack::HyperbolicTangentKernel h2(2.0, 1.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = h.Evaluate(x1, x2);
const double k2 = h2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (s=1.0, t=0.0), " << k2
    << " (s=2.0, t=1.0)." << std::endl;

// Change the scale and offset of the kernel.
h.Scale(2.5);
h.Offset(-1.0);
const double k3 = h.Evaluate(x1, x2);
std::cout << "Kernel value with s=2.5, t=-1.0: " << k3 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = h.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 1.5");
arma::fvec fx2("1.5 1.0 0.5");
const double k4 = h.Evaluate(fx1, fx2);
const double k5 = h2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k4
    << " (s=2.5, t=-1.0), " << k5 << " (s=2.0, t=1.0)." << std::endl;

🔗 LaplacianKernel

The LaplacianKernel class implements the Laplacian kernel, also known as the exponential kernel, defined by the following equation: k(x1, x2) = exp(-|| x1 - x2 || / bw) where bw is the bandwidth parameter.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Other utilities

🔗 Example usage

// Create a Laplacian kernel with default bandwidth.
mlpack::LaplacianKernel l;

// Create a Laplacian kernel with bandwidth 5.0.
mlpack::LaplacianKernel l2(5.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = l.Evaluate(x1, x2);
const double k2 = l2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (bw=1.0), " << k2 << " (bw=5.0)."
    << std::endl;

// Evaluate the kernel value when the distance between two points is already
// computed.
const double distance = 1.5;
const double k3 = l.Evaluate(distance);

// Change the bandwidth of the kernel to 2.5.
l.Bandwidth(2.5);
const double k4 = l.Evaluate(x1, x2);
std::cout << "Kernel value with bw=2.5: " << k4 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = l.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 1.5");
arma::fvec fx2("1.5 1.0 0.5");
const double k5 = l.Evaluate(fx1, fx2);
const double k6 = l2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k5
    << " (bw=2.5), " << k6 << " (bw=5.0)." << std::endl;

🔗 LinearKernel

The LinearKernel class implements the simple linear dot product kernel, defined by the following equation: k(x1, x2) = x1^T x2.

The use of the linear kernel for kernel methods generally results in the non-kernelized version of the algorithm; for instance, a kernel support vector machine using the linear kernel amounts to a linear SVM.

🔗 Constructor

Note: because the LinearKernel kernel has no parameters, it is not necessary to create an object and the Evaluate() function (below) can be called statically.

🔗 Kernel evaluation

🔗 Example usage

// Create a linear kernel.
mlpack::LinearKernel l;

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = l.Evaluate(x1, x2); // Identical to arma::dot(x1, x2).
const double k2 = l.Evaluate(x1, x1);
const double k3 = l.Evaluate(x2, x2);
std::cout << "Linear kernel values:" << std::endl;
std::cout << "  - k(x1, x2): " << k1 << "." << std::endl;
std::cout << "  - k(x1, x1): " << k2 << "." << std::endl;
std::cout << "  - k(x2, x2): " << k3 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix,
// using the static Evaluate() function.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = mlpack::LinearKernel::Evaluate(x1, r.col(i));
std::cout << "Average linear kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the cosine similarity between two sparse 32-bit floating point
// vectors.
arma::sp_fvec x3, x4;
x3.sprandu(100, 1, 0.2);
x4.sprandu(100, 1, 0.2);
const double k4 = mlpack::LinearKernel::Evaluate(x3, x4);
std::cout << "Linear kernel value between two random sparse 32-bit floating "
    << "point vectors: " << k4 << "." << std::endl;

🔗 PolynomialKernel

The PolynomialKernel class implements the standard polynomial kernel, which is defined by the following equation: k(x1, x2) = (x1^T x2 + t)^d where d is the degree of the polynomial and t is the offset.

The use of the polynomial kernel has a similar effect to the use of polynomial (interaction) features in standard machine learning methods.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Example usage

// Create a polynomial kernel with default degree (2) and offset (0).
mlpack::PolynomialKernel p;

// Create a polynomial kernel with degree 3.0 and offset -1.0.
mlpack::PolynomialKernel p2(3.0, -1.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = p.Evaluate(x1, x2);
const double k2 = p2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (bw=1.0), " << k2 << " (bw=5.0)."
    << std::endl;

// Change the degree of the kernel to 2.5 and the offset to 1.0.
p.Degree(2.5);
p.Offset(1.0);
const double k3 = p.Evaluate(x1, x2);
std::cout << "Kernel value with d=2.5, t=1.0: " << k3 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = p.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 1.5");
arma::fvec fx2("1.5 1.0 0.5");
const double k4 = p.Evaluate(fx1, fx2);
const double k5 = p2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k4
    << " (d=2.5, t=1.0), " << k5 << " (d=3.0, t=-1.0)." << std::endl;

🔗 PSpectrumStringKernel

The PSpectrumStringKernel class implements the length-p string spectrum kernel, proposed by Leslie, Eskin, and Noble (pdf). The kernel finds the contiguous subsequence match count between two strings.

Due to mlpack’s use of Armadillo, which requires that all matrix data be numeric, this class operates by internally storing all strings, and passing in numeric vectors such as [0 1] that reference string index 1 in dataset index 0. In turn, this means that the data points given to the PSpectrumStringKernel are simply IDs and have no geometric meaning.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Example usage

// Create two example datasets:
//      ["hello", "goodbye", "package"],
//      ["mlpack", "is", "really", "great"]
std::vector<std::vector<std::string>> datasets;
datasets.push_back({ "hello", "goodbye", "package" });
datasets.push_back({ "mlpack", "is", "really", "great" });

// Create a p-spectrum string kernel with a substring length of 2,
// and another with a substring length of 3.
mlpack::PSpectrumStringKernel p(datasets, 2);
mlpack::PSpectrumStringKernel p2(datasets, 3);

// Evaluate the kernel value between "mlpack" and "package".
arma::uvec x1("1 0"); // "mlpack": dataset 1, string 0.
arma::uvec x2("0 2"); // "package": dataset 0, string 2.
const double k1 = p.Evaluate(x1, x2);
const double k2 = p2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (p=2), " << k2 << " (p=3)."
    << std::endl;

🔗 SphericalKernel

The SphericalKernel class implements the simple spherical kernel, also known as the uniform kernel, or rectangular window kernel. The value of the SphericalKernel is 1 when the Euclidean distance between two points x1 and x2 is less than the bandwidth bw, and 0 otherwise: k(x1, x2) = 1(|| x1 - x2 || <= bw).

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Other utilities

🔗 Example usage

// Create a spherical kernel with default bandwidth.
mlpack::SphericalKernel s;

// Create a spherical kernel with bandwidth 5.0.
mlpack::SphericalKernel s2(5.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 2.5");
arma::vec x2("2.5 1.0 0.5");
const double k1 = s.Evaluate(x1, x2);
const double k2 = s2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (bw=1.0), " << k2 << " (bw=5.0)."
    << std::endl;

// Evaluate the kernel value when the distance between two points is already
// computed.
const double distance = 0.9;
const double k3 = s.Evaluate(distance);

// Change the bandwidth of the kernel to 3.0.
s.Bandwidth(3.0);
const double k4 = s.Evaluate(x1, x2);
std::cout << "Kernel value with bw=3.0: " << k4 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix, using
// a kernel bandwidth of 2.5.
s.Bandwidth(2.5);
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = s.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 2.5");
arma::fvec fx2("2.5 1.0 0.5");
const double k5 = s.Evaluate(fx1, fx2);
const double k6 = s2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k5
    << " (bw=2.5), " << k6 << " (bw=5.0)." << std::endl;

🔗 TriangularKernel

The TriangularKernel class implements the simple triangular kernel, defined by the following equation: k(x1, x2) = max(0, 1 - || x1 - x2 || / bw) where bw is the bandwidth of the kernel.

🔗 Constructors and properties

🔗 Kernel evaluation

🔗 Other utilities

🔗 Example usage

// Create a triangular kernel with default bandwidth.
mlpack::TriangularKernel t;

// Create a triangular kernel with bandwidth 5.0.
mlpack::TriangularKernel t2(5.0);

// Evaluate the kernel value between two 3-dimensional points.
arma::vec x1("0.5 1.0 1.5");
arma::vec x2("1.5 1.0 0.5");
const double k1 = t.Evaluate(x1, x2);
const double k2 = t2.Evaluate(x1, x2);
std::cout << "Kernel values: " << k1 << " (bw=1.0), " << k2 << " (bw=5.0)."
    << std::endl;

// Evaluate the kernel value when the distance between two points is already
// computed.
const double distance = 0.75;
const double k3 = t.Evaluate(distance);

// Change the bandwidth of the kernel to 2.5.
t.Bandwidth(2.5);
const double k4 = t.Evaluate(x1, x2);
std::cout << "Kernel value with bw=2.5: " << k4 << "." << std::endl;

// Evaluate the kernel value between x1 and all points in a random matrix.
arma::mat r(3, 100, arma::fill::randu);
arma::vec kernelValues(100);
for (size_t i = 0; i < r.n_cols; ++i)
  kernelValues[i] = t.Evaluate(x1, r.col(i));
std::cout << "Average kernel value for random points: "
    << arma::mean(kernelValues) << "." << std::endl;

// Compute the kernel value between two 32-bit floating-point vectors.
arma::fvec fx1("0.5 1.0 1.5");
arma::fvec fx2("1.5 1.0 0.5");
const double k5 = t.Evaluate(fx1, fx2);
const double k6 = t2.Evaluate(fx1, fx2);
std::cout << "Kernel values between two floating-point vectors: " << k5
    << " (bw=2.5), " << k6 << " (bw=5.0)." << std::endl;

🔗 Implement a custom kernel

mlpack supports custom kernels, so long as they implement an appropriate Evaluate() function.

See The KernelType Policy in mlpack for more information.