mlpack  git-master
AdaDeltaUpdate Class Reference

Implementation of the AdaDelta update policy. More...

Public Member Functions

 AdaDeltaUpdate (const double rho=0.95, const double epsilon=1e-6)
 Construct the AdaDelta update policy with given rho and epsilon parameters. More...

 
double Epsilon () const
 Get the value used to initialise the mean squared gradient parameter. More...

 
double & Epsilon ()
 Modify the value used to initialise the mean squared gradient parameter. More...

 
void Initialize (const size_t rows, const size_t cols)
 The Initialize method is called by SGD Optimizer method before the start of the iteration update process. More...

 
double Rho () const
 Get the smoothing parameter. More...

 
double & Rho ()
 Modify the smoothing parameter. More...

 
void Update (arma::mat &iterate, const double stepSize, const arma::mat &gradient)
 Update step for SGD. More...

 

Detailed Description

Implementation of the AdaDelta update policy.

AdaDelta is an optimizer that uses two ideas to improve upon the two main drawbacks of the AdaGrad method:

  • Accumulate Over Window
  • Correct Units with Hessian Approximation

For more information, see the following.

@article{Zeiler2012,
author = {Matthew D. Zeiler},
title = {{ADADELTA:} An Adaptive Learning Rate Method},
journal = {CoRR},
year = {2012}
}

Definition at line 40 of file ada_delta_update.hpp.

Constructor & Destructor Documentation

◆ AdaDeltaUpdate()

AdaDeltaUpdate ( const double  rho = 0.95,
const double  epsilon = 1e-6 
)
inline

Construct the AdaDelta update policy with given rho and epsilon parameters.

Parameters
rhoThe smoothing parameter.
epsilonThe epsilon value used to initialise the squared gradient parameter.

Definition at line 50 of file ada_delta_update.hpp.

Member Function Documentation

◆ Epsilon() [1/2]

double Epsilon ( ) const
inline

Get the value used to initialise the mean squared gradient parameter.

Definition at line 107 of file ada_delta_update.hpp.

◆ Epsilon() [2/2]

double& Epsilon ( )
inline

Modify the value used to initialise the mean squared gradient parameter.

Definition at line 109 of file ada_delta_update.hpp.

◆ Initialize()

void Initialize ( const size_t  rows,
const size_t  cols 
)
inline

The Initialize method is called by SGD Optimizer method before the start of the iteration update process.

In AdaDelta update policy, the mean squared and the delta mean squared gradient matrices are initialized to the zeros matrix with the same size as gradient matrix (see mlpack::optimization::SGD::Optimizer).

Parameters
rowsNumber of rows in the gradient matrix.
colsNumber of columns in the gradient matrix.

Definition at line 67 of file ada_delta_update.hpp.

◆ Rho() [1/2]

double Rho ( ) const
inline

Get the smoothing parameter.

Definition at line 102 of file ada_delta_update.hpp.

◆ Rho() [2/2]

double& Rho ( )
inline

Modify the smoothing parameter.

Definition at line 104 of file ada_delta_update.hpp.

◆ Update()

void Update ( arma::mat &  iterate,
const double  stepSize,
const arma::mat &  gradient 
)
inline

Update step for SGD.

The AdaDelta update dynamically adapts over time using only first order information. Additionally, AdaDelta requires no manual tuning of a learning rate.

Parameters
iterateParameters that minimize the function.
stepSizeStep size to be used for the given iteration.
gradientThe gradient matrix.

Definition at line 83 of file ada_delta_update.hpp.


The documentation for this class was generated from the following file: