mlpack  git-master
GradientDescent Class Reference

Gradient Descent is a technique to minimize a function. More...

Public Member Functions

 GradientDescent (const double stepSize=0.01, const size_t maxIterations=100000, const double tolerance=1e-5)
 Construct the Gradient Descent optimizer with the given function and parameters. More...

 
size_t MaxIterations () const
 Get the maximum number of iterations (0 indicates no limit). More...

 
size_t & MaxIterations ()
 Modify the maximum number of iterations (0 indicates no limit). More...

 
template
<
typename
FunctionType
>
double Optimize (FunctionType &function, arma::mat &iterate)
 Optimize the given function using gradient descent. More...

 
template
<
typename
FunctionType
>
double Optimize (FunctionType &function, arma::mat &iterate, data::DatasetMapper< data::IncrementPolicy, double > &datasetInfo)
 Assert all dimensions are numeric and optimize the given function using gradient descent. More...

 
double StepSize () const
 Get the step size. More...

 
double & StepSize ()
 Modify the step size. More...

 
double Tolerance () const
 Get the tolerance for termination. More...

 
double & Tolerance ()
 Modify the tolerance for termination. More...

 

Detailed Description

Gradient Descent is a technique to minimize a function.

To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient of the function at the current point, producing the following update scheme:

\[ A_{j + 1} = A_j + \alpha \nabla F(A) \]

where $ \alpha $ is a parameter which specifies the step size. $ F $ is the function being optimized. The algorithm continues until $ j $ reaches the maximum number of iterations—or when an update produces an improvement within a certain tolerance $ \epsilon $. That is,

\[ | F(A_{j + 1}) - F(A_j) | < \epsilon. \]

The parameter $\epsilon$ is specified by the tolerance parameter to the constructor.

For Gradient Descent to work, a FunctionType template parameter is required. This class must implement the following function:

double Evaluate(const arma::mat& coordinates); void Gradient(const arma::mat& coordinates, arma::mat& gradient);

Definition at line 49 of file gradient_descent.hpp.

Constructor & Destructor Documentation

◆ GradientDescent()

GradientDescent ( const double  stepSize = 0.01,
const size_t  maxIterations = 100000,
const double  tolerance = 1e-5 
)

Construct the Gradient Descent optimizer with the given function and parameters.

The defaults here are not necessarily good for the given problem, so it is suggested that the values used be tailored to the task at hand.

Parameters
functionFunction to be optimized (minimized).
stepSizeStep size for each iteration.
maxIterationsMaximum number of iterations allowed (0 means no limit).
toleranceMaximum absolute tolerance to terminate algorithm.

Member Function Documentation

◆ MaxIterations() [1/2]

size_t MaxIterations ( ) const
inline

Get the maximum number of iterations (0 indicates no limit).

Definition at line 108 of file gradient_descent.hpp.

◆ MaxIterations() [2/2]

size_t& MaxIterations ( )
inline

Modify the maximum number of iterations (0 indicates no limit).

Definition at line 110 of file gradient_descent.hpp.

◆ Optimize() [1/2]

double Optimize ( FunctionType &  function,
arma::mat &  iterate 
)

Optimize the given function using gradient descent.

The given starting point will be modified to store the finishing point of the algorithm, and the final objective value is returned.

Template Parameters
FunctionTypeType of the function to optimize.
Parameters
functionFunction to optimize.
iterateStarting point (will be modified).
Returns
Objective value of the final point.

◆ Optimize() [2/2]

double Optimize ( FunctionType &  function,
arma::mat &  iterate,
data::DatasetMapper< data::IncrementPolicy, double > &  datasetInfo 
)

Assert all dimensions are numeric and optimize the given function using gradient descent.

The given starting point will be modified to store the finishing point of the algorithm, and the final objective value is returned.

This overload is intended to be used primarily by the hyper-parameter tuning module.

Template Parameters
FunctionTypeType of the function to optimize.
Parameters
functionFunction to optimize.
iterateStarting point (will be modified).
datasetInfoType information for each dimension of the dataset.
Returns
Objective value of the final point.

◆ StepSize() [1/2]

double StepSize ( ) const
inline

Get the step size.

Definition at line 103 of file gradient_descent.hpp.

◆ StepSize() [2/2]

double& StepSize ( )
inline

Modify the step size.

Definition at line 105 of file gradient_descent.hpp.

◆ Tolerance() [1/2]

double Tolerance ( ) const
inline

Get the tolerance for termination.

Definition at line 113 of file gradient_descent.hpp.

◆ Tolerance() [2/2]

double& Tolerance ( )
inline

Modify the tolerance for termination.

Definition at line 115 of file gradient_descent.hpp.


The documentation for this class was generated from the following file: