WeightNorm< InputDataType, OutputDataType, CustomLayers > Class Template Reference

Declaration of the WeightNorm layer class. More...

Public Member Functions

 WeightNorm (LayerTypes< CustomLayers... > layer=LayerTypes< CustomLayers... >())
 Create the WeightNorm layer object. More...

 
 ~WeightNorm ()
 Destructor to release allocated memory. More...

 
template
<
typename
eT
>
void Backward (const arma::Mat< eT > &input, const arma::Mat< eT > &gy, arma::Mat< eT > &g)
 Backward pass through the layer. More...

 
OutputDataType const & Delta () const
 Get the delta. More...

 
OutputDataType & Delta ()
 Modify the delta. More...

 
template
<
typename
eT
>
void Forward (const arma::Mat< eT > &input, arma::Mat< eT > &output)
 Forward pass of the WeightNorm layer. More...

 
template
<
typename
eT
>
void Gradient (const arma::Mat< eT > &input, const arma::Mat< eT > &error, arma::Mat< eT > &gradient)
 Calculate the gradient using the output delta, input activations and the weights of the wrapped layer. More...

 
OutputDataType const & Gradient () const
 Get the gradient. More...

 
OutputDataType & Gradient ()
 Modify the gradient. More...

 
LayerTypes< CustomLayers... > const & Layer ()
 Get the wrapped layer. More...

 
OutputDataType const & OutputParameter () const
 Get the output parameter. More...

 
OutputDataType & OutputParameter ()
 Modify the output parameter. More...

 
OutputDataType const & Parameters () const
 Get the parameters. More...

 
OutputDataType & Parameters ()
 Modify the parameters. More...

 
void Reset ()
 Reset the layer parameters. More...

 
template
<
typename
Archive
>
void serialize (Archive &ar, const uint32_t)
 Serialize the layer. More...

 

Detailed Description


template<typename InputDataType = arma::mat, typename OutputDataType = arma::mat, typename... CustomLayers>
class mlpack::ann::WeightNorm< InputDataType, OutputDataType, CustomLayers >

Declaration of the WeightNorm layer class.

The layer reparameterizes the weight vectors in a neural network, decoupling the length of those weight vectors from their direction. This reparameterization does not introduce any dependencies between the examples in a mini-batch.

This class will be a wrapper around existing layers. It will just modify the calculation and updation of weights of the layer.

For more information, refer to the following paper,

@inproceedings{Salimans2016WeightNorm,
title = {Weight Normalization: A Simple Reparameterization to Accelerate
Training of Deep Neural Networks},
author = {Tim Salimans, Diederik P. Kingma},
booktitle = {Neural Information Processing Systems 2016},
year = {2016},
url = {https://arxiv.org/abs/1602.07868},
}
Template Parameters
InputDataTypeType of the input data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
OutputDataTypeType of the output data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
CustomLayersAdditional custom layers that can be added.

Definition at line 217 of file layer_types.hpp.

Constructor & Destructor Documentation

◆ WeightNorm()

WeightNorm ( LayerTypes< CustomLayers... >  layer = LayerTypes< CustomLayers... >())

Create the WeightNorm layer object.

Parameters
layerThe layer whose weights are needed to be normalized.

◆ ~WeightNorm()

~WeightNorm ( )

Destructor to release allocated memory.

Member Function Documentation

◆ Backward()

void Backward ( const arma::Mat< eT > &  input,
const arma::Mat< eT > &  gy,
arma::Mat< eT > &  g 
)

Backward pass through the layer.

This function calls the Backward() function of the wrapped layer.

Parameters
inputThe input activations.
gyThe backpropagated error.
gThe calculated gradient.

◆ Delta() [1/2]

OutputDataType const& Delta ( ) const
inline

Get the delta.

Definition at line 118 of file weight_norm.hpp.

◆ Delta() [2/2]

OutputDataType& Delta ( )
inline

Modify the delta.

Definition at line 120 of file weight_norm.hpp.

◆ Forward()

void Forward ( const arma::Mat< eT > &  input,
arma::Mat< eT > &  output 
)

Forward pass of the WeightNorm layer.

Calculates the weights of the wrapped layer from the parameter vector v and the scalar parameter g. It then calulates the output of the wrapped layer from the calculated weights.

Parameters
inputInput data for the layer.
outputResulting output activations.

◆ Gradient() [1/3]

void Gradient ( const arma::Mat< eT > &  input,
const arma::Mat< eT > &  error,
arma::Mat< eT > &  gradient 
)

Calculate the gradient using the output delta, input activations and the weights of the wrapped layer.

Parameters
inputThe input activations.
errorThe calculated error.
gradientThe calculated gradient.

◆ Gradient() [2/3]

OutputDataType const& Gradient ( ) const
inline

Get the gradient.

Definition at line 123 of file weight_norm.hpp.

◆ Gradient() [3/3]

OutputDataType& Gradient ( )
inline

Modify the gradient.

Definition at line 125 of file weight_norm.hpp.

◆ Layer()

LayerTypes<CustomLayers...> const& Layer ( )
inline

Get the wrapped layer.

Definition at line 138 of file weight_norm.hpp.

References WeightNorm< InputDataType, OutputDataType, CustomLayers >::serialize().

◆ OutputParameter() [1/2]

OutputDataType const& OutputParameter ( ) const
inline

Get the output parameter.

Definition at line 128 of file weight_norm.hpp.

◆ OutputParameter() [2/2]

OutputDataType& OutputParameter ( )
inline

Modify the output parameter.

Definition at line 130 of file weight_norm.hpp.

◆ Parameters() [1/2]

OutputDataType const& Parameters ( ) const
inline

Get the parameters.

Definition at line 133 of file weight_norm.hpp.

◆ Parameters() [2/2]

OutputDataType& Parameters ( )
inline

Modify the parameters.

Definition at line 135 of file weight_norm.hpp.

◆ Reset()

void Reset ( )

Reset the layer parameters.

◆ serialize()

void serialize ( Archive &  ar,
const uint32_t   
)

The documentation for this class was generated from the following files:
  • /home/jenkins-mlpack/mlpack.org/_src/mlpack-git/src/mlpack/methods/ann/layer/layer_types.hpp
  • /home/jenkins-mlpack/mlpack.org/_src/mlpack-git/src/mlpack/methods/ann/layer/weight_norm.hpp