mlpack  git-master
OptimisticAdamUpdate Class Reference

OptimisticAdam is an optimizer which implements the Optimistic Adam algorithm which uses Optmistic Mirror Descent with the Adam Optimizer. More...

Public Member Functions

 OptimisticAdamUpdate (const double epsilon=1e-8, const double beta1=0.9, const double beta2=0.999)
 Construct the OptimisticAdam update policy with the given parameters. More...

 
double Beta1 () const
 Get the smoothing parameter. More...

 
double & Beta1 ()
 Modify the smoothing parameter. More...

 
double Beta2 () const
 Get the second moment coefficient. More...

 
double & Beta2 ()
 Modify the second moment coefficient. More...

 
double Epsilon () const
 Get the value used to initialize the squared gradient parameter. More...

 
double & Epsilon ()
 Modify the value used to initialize the squared gradient parameter. More...

 
void Initialize (const size_t rows, const size_t cols)
 The Initialize method is called by SGD Optimizer method before the start of the iteration update process. More...

 
void Update (arma::mat &iterate, const double stepSize, const arma::mat &gradient)
 Update step for OptimisticAdam. More...

 

Detailed Description

OptimisticAdam is an optimizer which implements the Optimistic Adam algorithm which uses Optmistic Mirror Descent with the Adam Optimizer.

It addresses the problem of limit cycling while training GANs. It uses OMD to achieve faster regret rates in solving the zero sum game of training a GAN. It consistently achieves a smaller KL divergnce with respect to the true underlying data distribution.

For more information, see the following.

@article{
author = {Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis,
Haoyang Zeng},
title = {Training GANs with Optimism},
year = {2017},
url = {https://arxiv.org/abs/1711.00141}
}

Definition at line 41 of file optimisticadam_update.hpp.

Constructor & Destructor Documentation

◆ OptimisticAdamUpdate()

OptimisticAdamUpdate ( const double  epsilon = 1e-8,
const double  beta1 = 0.9,
const double  beta2 = 0.999 
)
inline

Construct the OptimisticAdam update policy with the given parameters.

Parameters
epsilonThe epsilon value used to initialize the squared gradient parameter.
beta1The smoothing parameter.
beta2The second moment coefficient.

Definition at line 52 of file optimisticadam_update.hpp.

Member Function Documentation

◆ Beta1() [1/2]

double Beta1 ( ) const
inline

Get the smoothing parameter.

Definition at line 114 of file optimisticadam_update.hpp.

◆ Beta1() [2/2]

double& Beta1 ( )
inline

Modify the smoothing parameter.

Definition at line 116 of file optimisticadam_update.hpp.

◆ Beta2() [1/2]

double Beta2 ( ) const
inline

Get the second moment coefficient.

Definition at line 119 of file optimisticadam_update.hpp.

◆ Beta2() [2/2]

double& Beta2 ( )
inline

Modify the second moment coefficient.

Definition at line 121 of file optimisticadam_update.hpp.

◆ Epsilon() [1/2]

double Epsilon ( ) const
inline

Get the value used to initialize the squared gradient parameter.

Definition at line 109 of file optimisticadam_update.hpp.

◆ Epsilon() [2/2]

double& Epsilon ( )
inline

Modify the value used to initialize the squared gradient parameter.

Definition at line 111 of file optimisticadam_update.hpp.

◆ Initialize()

void Initialize ( const size_t  rows,
const size_t  cols 
)
inline

The Initialize method is called by SGD Optimizer method before the start of the iteration update process.

Parameters
rowsNumber of rows in the gradient matrix.
colsNumber of columns in the gradient matrix.

Definition at line 70 of file optimisticadam_update.hpp.

◆ Update()

void Update ( arma::mat &  iterate,
const double  stepSize,
const arma::mat &  gradient 
)
inline

Update step for OptimisticAdam.

Parameters
iterateParameters that minimize the function.
stepSizeStep size to be used for the given iteration.
gradientThe gradient matrix.

Definition at line 84 of file optimisticadam_update.hpp.


The documentation for this class was generated from the following file: