WeightNorm< InputDataType, OutputDataType, CustomLayers > Class Template Reference

Declaration of the WeightNorm layer class. More...

Public Member Functions
	WeightNorm (LayerTypes< CustomLayers... > layer=LayerTypes< CustomLayers... >())
	Create the WeightNorm layer object. More...
	~WeightNorm ()
	Destructor to release allocated memory. More...
template < typename eT >
void	Backward (const arma::Mat< eT > &&input, arma::Mat< eT > &&gy, arma::Mat< eT > &&g)
	Backward pass through the layer. More...
OutputDataType const &	Delta () const
	Get the delta. More...
OutputDataType &	Delta ()
	Modify the delta. More...
template < typename eT >
void	Forward (arma::Mat< eT > &&input, arma::Mat< eT > &&output)
	Forward pass of the WeightNorm layer. More...
template < typename eT >
void	Gradient (arma::Mat< eT > &&input, arma::Mat< eT > &&error, arma::Mat< eT > &&gradient)
	Calculate the gradient using the output delta, input activations and the weights of the wrapped layer. More...
OutputDataType const &	Gradient () const
	Get the gradient. More...
OutputDataType &	Gradient ()
	Modify the gradient. More...
LayerTypes< CustomLayers... > const &	Layer ()
	Get the wrapped layer. More...
OutputDataType const &	OutputParameter () const
	Get the output parameter. More...
OutputDataType &	OutputParameter ()
	Modify the output parameter. More...
OutputDataType const &	Parameters () const
	Get the parameters. More...
OutputDataType &	Parameters ()
	Modify the parameters. More...
void	Reset ()
	Reset the layer parameters. More...
template < typename Archive >
void	serialize (Archive &ar, const unsigned int)
	Serialize the layer. More...

Detailed Description

template<typename InputDataType = arma::mat, typename OutputDataType = arma::mat, typename... CustomLayers>
class mlpack::ann::WeightNorm< InputDataType, OutputDataType, CustomLayers >

Declaration of the WeightNorm layer class.

The layer reparameterizes the weight vectors in a neural network, decoupling the length of those weight vectors from their direction. This reparameterization does not introduce any dependencies between the examples in a mini-batch.

This class will be a wrapper around existing layers. It will just modify the calculation and updation of weights of the layer.

For more information, refer to the following paper,

@inproceedings{Salimans2016WeightNorm,
  title = {Weight Normalization: A Simple Reparameterization to Accelerate
           Training of Deep Neural Networks},
  author = {Tim Salimans, Diederik P. Kingma},
  booktitle = {Neural Information Processing Systems 2016},
  year = {2016}
}

Template Parameters

InputDataType	Type of the input data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
OutputDataType	Type of the output data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
CustomLayers	Additional custom layers that can be added.

Definition at line 176 of file layer_types.hpp.

Constructor & Destructor Documentation

WeightNorm()

WeightNorm

(

LayerTypes< CustomLayers... >

layer = LayerTypes< CustomLayers... >()

)

Create the WeightNorm layer object.

Parameters

layer

The layer whose weights are needed to be normalized.

~WeightNorm()

~WeightNorm

(

)

Destructor to release allocated memory.

Member Function Documentation

Backward()

void Backward	(	const arma::Mat< eT > &&	input,
		arma::Mat< eT > &&	gy,
		arma::Mat< eT > &&	g
	)

Backward pass through the layer.

This function calls the Backward() function of the wrapped layer.

Parameters

input	The input activations.
gy	The backpropagated error.
g	The calculated gradient.

Delta() [1/2]

OutputDataType const& Delta ( ) const

inline

Get the delta.

Definition at line 117 of file weight_norm.hpp.

Delta() [2/2]

OutputDataType& Delta ( )

inline

Modify the delta.

Definition at line 119 of file weight_norm.hpp.

Forward()

void Forward	(	arma::Mat< eT > &&	input,
		arma::Mat< eT > &&	output
	)

Forward pass of the WeightNorm layer.

Calculates the weights of the wrapped layer from the parameter vector v and the scalar parameter g. It then calulates the output of the wrapped layer from the calculated weights.

Parameters

input	Input data for the layer.
output	Resulting output activations.

Gradient() [1/3]

void Gradient	(	arma::Mat< eT > &&	input,
		arma::Mat< eT > &&	error,
		arma::Mat< eT > &&	gradient
	)

Calculate the gradient using the output delta, input activations and the weights of the wrapped layer.

Parameters

input	The input activations.
error	The calculated error.
gradient	The calculated gradient.

Gradient() [2/3]

OutputDataType const& Gradient ( ) const

inline

Get the gradient.

Definition at line 122 of file weight_norm.hpp.

Gradient() [3/3]

OutputDataType& Gradient ( )

inline

Modify the gradient.

Definition at line 124 of file weight_norm.hpp.

Layer()

LayerTypes<CustomLayers...> const& Layer ( )

inline

Get the wrapped layer.

Definition at line 137 of file weight_norm.hpp.

References WeightNorm< InputDataType, OutputDataType, CustomLayers >::serialize().

OutputParameter() [1/2]

OutputDataType const& OutputParameter ( ) const

inline

Get the output parameter.

Definition at line 127 of file weight_norm.hpp.

OutputParameter() [2/2]

OutputDataType& OutputParameter ( )

inline

Modify the output parameter.

Definition at line 129 of file weight_norm.hpp.

Parameters() [1/2]

OutputDataType const& Parameters ( ) const

inline

Get the parameters.

Definition at line 132 of file weight_norm.hpp.

Parameters() [2/2]

OutputDataType& Parameters ( )

inline

Modify the parameters.

Definition at line 134 of file weight_norm.hpp.

Reset()

void Reset

(

)

Reset the layer parameters.

serialize()

void serialize	(	Archive &	ar,
		const unsigned	int
	)

Serialize the layer.

Referenced by WeightNorm< InputDataType, OutputDataType, CustomLayers >::Layer().

---

The documentation for this class was generated from the following files:

• /home/jenkins-mlpack/mlpack.org/_src/mlpack-3.2.1/src/mlpack/methods/ann/layer/layer_types.hpp
• /home/jenkins-mlpack/mlpack.org/_src/mlpack-3.2.1/src/mlpack/methods/ann/layer/weight_norm.hpp