mlpack  git-master
NadamUpdate Class Reference

Nadam is an optimizer that combines the Adam and NAG optimization strategies. More...

Public Member Functions

 NadamUpdate (const double epsilon=1e-8, const double beta1=0.9, const double beta2=0.99, const double scheduleDecay=4e-3)
 Construct the Nadam update policy with the given parameters. More...

 
double Beta1 () const
 Get the smoothing parameter. More...

 
double & Beta1 ()
 Modify the smoothing parameter. More...

 
double Beta2 () const
 Get the second moment coefficient. More...

 
double & Beta2 ()
 Modify the second moment coefficient. More...

 
double CumBeta1 () const
 Get the value of the cumulative product of decay coefficients. More...

 
double & CumBeta1 ()
 Modify the value of the cumulative product of decay coefficients. More...

 
double Epsilon () const
 Get the value used to initialise the squared gradient parameter. More...

 
double & Epsilon ()
 Modify the value used to initialise the squared gradient parameter. More...

 
void Initialize (const size_t rows, const size_t cols)
 The Initialize() method is called by the optimizer before the start of the iteration update process. More...

 
double ScheduleDecay () const
 Get the decay parameter for decay coefficients. More...

 
double & ScheduleDecay ()
 Modify the decay parameter for decay coefficients. More...

 
void Update (arma::mat &iterate, const double stepSize, const arma::mat &gradient)
 Update step for Nadam. More...

 

Detailed Description

Nadam is an optimizer that combines the Adam and NAG optimization strategies.

For more information, see the following.

@techreport{Dozat2015,
title = {Incorporating Nesterov momentum into Adam},
author = {Timothy Dozat},
institution = {Stanford University},
address = {Stanford},
year = {2015},
url = {https://openreview.net/pdf?id=OM0jvwB8jIp57ZJjtNEZ}
}

Definition at line 37 of file nadam_update.hpp.

Constructor & Destructor Documentation

◆ NadamUpdate()

NadamUpdate ( const double  epsilon = 1e-8,
const double  beta1 = 0.9,
const double  beta2 = 0.99,
const double  scheduleDecay = 4e-3 
)
inline

Construct the Nadam update policy with the given parameters.

Parameters
epsilonThe epsilon value used to initialise the squared gradient parameter.
beta1The smoothing parameter.
beta2The second moment coefficient
scheduleDecayThe decay parameter for decay coefficients

Definition at line 49 of file nadam_update.hpp.

Member Function Documentation

◆ Beta1() [1/2]

double Beta1 ( ) const
inline

Get the smoothing parameter.

Definition at line 130 of file nadam_update.hpp.

◆ Beta1() [2/2]

double& Beta1 ( )
inline

Modify the smoothing parameter.

Definition at line 132 of file nadam_update.hpp.

◆ Beta2() [1/2]

double Beta2 ( ) const
inline

Get the second moment coefficient.

Definition at line 135 of file nadam_update.hpp.

◆ Beta2() [2/2]

double& Beta2 ( )
inline

Modify the second moment coefficient.

Definition at line 137 of file nadam_update.hpp.

◆ CumBeta1() [1/2]

double CumBeta1 ( ) const
inline

Get the value of the cumulative product of decay coefficients.

Definition at line 125 of file nadam_update.hpp.

◆ CumBeta1() [2/2]

double& CumBeta1 ( )
inline

Modify the value of the cumulative product of decay coefficients.

Definition at line 127 of file nadam_update.hpp.

◆ Epsilon() [1/2]

double Epsilon ( ) const
inline

Get the value used to initialise the squared gradient parameter.

Definition at line 120 of file nadam_update.hpp.

◆ Epsilon() [2/2]

double& Epsilon ( )
inline

Modify the value used to initialise the squared gradient parameter.

Definition at line 122 of file nadam_update.hpp.

◆ Initialize()

void Initialize ( const size_t  rows,
const size_t  cols 
)
inline

The Initialize() method is called by the optimizer before the start of the iteration update process.

Parameters
rowsNumber of rows in the gradient matrix.
colsNumber of columns in the gradient matrix.

Definition at line 70 of file nadam_update.hpp.

◆ ScheduleDecay() [1/2]

double ScheduleDecay ( ) const
inline

Get the decay parameter for decay coefficients.

Definition at line 140 of file nadam_update.hpp.

◆ ScheduleDecay() [2/2]

double& ScheduleDecay ( )
inline

Modify the decay parameter for decay coefficients.

Definition at line 142 of file nadam_update.hpp.

◆ Update()

void Update ( arma::mat &  iterate,
const double  stepSize,
const arma::mat &  gradient 
)
inline

Update step for Nadam.

Parameters
iterateParameters that minimize the function.
stepSizeStep size to be used for the given iteration.
gradientThe gradient matrix.

Definition at line 83 of file nadam_update.hpp.


The documentation for this class was generated from the following file: