Layer< MatType > Class Template Referenceabstract

A layer is an abstract class implementing common neural networks operations, such as convolution, batch norm, etc. More...

Inheritance diagram for Layer< MatType >:

Public Member Functions

 Layer ()
 Default constructor. More...

 
 Layer (const Layer &layer)
 Copy constructor. This is not responsible for copying weights! More...

 
 Layer (Layer &&layer)
 Move constructor. This is not responsible for moving weights! More...

 
virtual ~Layer ()
 Default deconstructor. More...

 
virtual void Backward (const MatType &, const MatType &, MatType &)
 Performs a backpropagation step through the layer, with respect to the given input. More...

 
virtual LayerClone () const =0
 Make a copy of the object. More...

 
virtual void ComputeOutputDimensions ()
 Compute the output dimensions. More...

 
virtual void Forward (const MatType &, MatType &)
 Takes an input object, and computes the corresponding output of the layer. More...

 
virtual void Forward (const MatType &, const MatType &)
 Takes an input and output object, and computes the corresponding loss of the layer. More...

 
virtual void Gradient (const MatType &, const MatType &, MatType &)
 Computing the gradient of the layer with respect to its own input. More...

 
const std::vector< size_t > & InputDimensions () const
 Get the input dimensions. More...

 
std::vector< size_t > & InputDimensions ()
 Modify the input dimensions. More...

 
virtual double Loss ()
 Get the layer loss. More...

 
virtual Layeroperator= (const Layer &layer)
 Copy assignment operator. This is not responsible for copying weights! More...

 
virtual Layeroperator= (Layer &&layer)
 Move assignment operator. This is not responsible for moving weights! More...

 
const std::vector< size_t > & OutputDimensions ()
 Get the output dimensions. More...

 
virtual size_t OutputSize () final
 Get the number of elements in the output from this layer. More...

 
virtual const MatType & Parameters () const
 Get the parameters. More...

 
virtual MatType & Parameters ()
 Set the parameters. More...

 
template
<
typename
Archive
>
void serialize (Archive &ar, const uint32_t)
 Serialize the layer. More...

 
virtual void SetWeights (typename MatType::elem_type *)
 Reset the layer parameter. More...

 
virtual bool const & Training () const
 Get whether the layer is currently in training mode. More...

 
virtual bool & Training ()
 Modify whether the layer is currently in training mode. More...

 
virtual size_t WeightSize () const
 Get the total number of trainable weights in the layer. More...

 

Protected Attributes

std::vector< size_t > inputDimensions
 Logical input dimensions of each point. More...

 
std::vector< size_t > outputDimensions
 Logical output dimensions of each point. More...

 
bool training
 If true, the layer is in training mode; otherwise, it is in testing mode. More...

 
bool validOutputDimensions
 This is true if ComputeOutputDimensions() has been called, and outputDimensions can be considered to be up-to-date. More...

 

Detailed Description


template
<
typename
MatType
=
arma::mat
>

class mlpack::ann::Layer< MatType >

A layer is an abstract class implementing common neural networks operations, such as convolution, batch norm, etc.

These operations require managing weights, losses, updates, and inter-layer connectivity.

Users will just instantiate a layer by inherited from the abstract class and implement the layer specific methods. It is recommend that descendants of Layer implement the following methods:

  • Constructor: Defines custom layer attributes, and creates layer state variables.
  • Forward(input, output): Performs the forward logic of applying the layer to the input object and storing the result in the output object.
  • Backward(input, gy, g): Performs a backpropagation step through the layer, with respect to the given input.
  • Gradient(input, error, gradient): Computing the gradient of the layer with respect to its own input.

The memory for the layer's parameters (weights and biases) is not allocated by the layer itself, instead it is allocated by the network that the layer belongs to, and passed to the layer when it needs to use it.

See the linear layer implementation for a basic example. It's a layer with two variables, w and b, that returns y = w * x + b. It shows how to implement Forward(), Backward() and Gradient(). The weights of the layers are tracked in layer.Parameters().

Template Parameters
MatTypeMatrix representation to accept as input and use for computation.

Definition at line 52 of file layer.hpp.

Constructor & Destructor Documentation

◆ Layer() [1/3]

Layer ( )
inline

Default constructor.

Definition at line 56 of file layer.hpp.

◆ ~Layer()

virtual ~Layer ( )
inlinevirtual

Default deconstructor.

Definition at line 60 of file layer.hpp.

◆ Layer() [2/3]

Layer ( const Layer< MatType > &  layer)
inline

Copy constructor. This is not responsible for copying weights!

Definition at line 63 of file layer.hpp.

◆ Layer() [3/3]

Layer ( Layer< MatType > &&  layer)
inline

Move constructor. This is not responsible for moving weights!

Definition at line 74 of file layer.hpp.

Member Function Documentation

◆ Backward()

virtual void Backward ( const MatType &  ,
const MatType &  ,
MatType &   
)
inlinevirtual

Performs a backpropagation step through the layer, with respect to the given input.

In general this method makes the assumption Forward(input, output) has been called before, with the same input. If you do not respect this rule, Backward(input, gy, g) might compute incorrect results.

In general input and gy and g are matrices. However, some special sub-classes like table layers might expect something else. Please, refer to each module specification for further information.

A backpropagation step consist of computing of computing the gradient output input with respect to the output of the layer and given error.

During the backward pass our goal is to use 'gy' in order to compute the downstream gradients (g). We assume that the upstream gradient (gy) has already been computed and is passed to the layer.

Parameters
*(input) The propagated input activation.
*(gy) The backpropagated error.
*(g) The calculated gradient.

Reimplemented in ConvolutionType< ForwardConvolutionRule, BackwardConvolutionRule, GradientConvolutionRule, MatType >, MaxPoolingType< MatType >, LSTMType< MatType >, MultiLayer< MatType >, MultiLayer< CustomLayers... >, MultiLayer< InputType, OutputType >, BaseLayer< ActivationFunction, MatType >, LinearType< MatType, RegularizerType >, DropConnectType< MatType >, RBFType< MatType, Activation >, AlphaDropoutType< MatType >, Linear3DType< MatType, RegularizerType >, AdaptiveMeanPoolingType< MatType >, LinearNoBiasType< MatType, RegularizerType >, AdaptiveMaxPoolingType< MatType >, DropoutType< MatType >, ConcatType< MatType >, LeakyReLUType< MatType >, MeanPoolingType< MatType >, NoisyLinearType< MatType >, PaddingType< MatType >, PaddingType< arma::mat >, PaddingType< InputType, OutputType >, ConcatenateType< MatType >, LogSoftMaxType< MatType >, SoftmaxType< MatType >, SoftmaxType< arma::mat >, and AddType< MatType >.

Definition at line 156 of file layer.hpp.

◆ Clone()

virtual Layer* Clone ( ) const
pure virtual

Make a copy of the object.

Implemented in ConvolutionType< ForwardConvolutionRule, BackwardConvolutionRule, GradientConvolutionRule, MatType >, ELUType< InputType, OutputType >, FastLSTMType< InputType, OutputType >, MaxPoolingType< MatType >, MultiheadAttentionType< InputType, OutputType, RegularizerType >, SequentialType< InputType, OutputType, Residual >, SequentialType< InputType, OutputType >, BaseLayer< ActivationFunction, MatType >, LayerNormType< InputType, OutputType >, WeightNormType< InputType, OutputType >, ReparametrizationType< InputType, OutputType >, LSTMType< MatType >, CELUType< InputType, OutputType >, FlexibleReLUType< InputType, OutputType >, VirtualBatchNormType< InputType, OutputType >, AdaptiveMeanPoolingType< MatType >, RBFType< MatType, Activation >, AdaptiveMaxPoolingType< MatType >, DropConnectType< MatType >, MultiplyMergeType< InputType, OutputType >, SoftShrinkType< InputType, OutputType >, MeanPoolingType< MatType >, SpatialDropoutType< InputType, OutputType >, AlphaDropoutType< MatType >, HardTanHType< InputType, OutputType >, HardShrinkType< InputType, OutputType >, LinearType< MatType, RegularizerType >, HighwayType< InputType, OutputType >, ConstantType< InputType, OutputType >, MultiLayer< MatType >, SubviewType< InputType, OutputType >, MultiLayer< CustomLayers... >, MultiLayer< InputType, OutputType >, DropoutType< MatType >, PReLUType< InputType, OutputType >, ConcatType< MatType >, CReLUType< InputType, OutputType >, PositionalEncodingType< InputType, OutputType >, Linear3DType< MatType, RegularizerType >, LeakyReLUType< MatType >, LinearNoBiasType< MatType, RegularizerType >, LookupType< InputType, OutputType >, SelectType< InputType, OutputType >, ReinforceNormalType< InputType, OutputType >, PaddingType< MatType >, PaddingType< arma::mat >, PaddingType< InputType, OutputType >, ConcatenateType< MatType >, JoinType< InputType, OutputType >, LogSoftMaxType< MatType >, MultiplyConstantType< InputType, OutputType >, SoftminType< InputType, OutputType >, SoftmaxType< MatType >, SoftmaxType< arma::mat >, NoisyLinearType< MatType >, and AddType< MatType >.

Referenced by Layer< InputDataType, OutputDataType >::Layer().

◆ ComputeOutputDimensions()

◆ Forward() [1/2]

◆ Forward() [2/2]

virtual void Forward ( const MatType &  ,
const MatType &   
)
inlinevirtual

Takes an input and output object, and computes the corresponding loss of the layer.

In general input and output are matrices. However, some special layers like table layers might expect something else. Please, refer to each layer specification for further information.

Parameters
*(input) Input data used for evaluating the specified layer.
*(output) Resulting output.

Definition at line 131 of file layer.hpp.

◆ Gradient()

virtual void Gradient ( const MatType &  ,
const MatType &  ,
MatType &   
)
inlinevirtual

Computing the gradient of the layer with respect to its own input.

This is returned in gradient.

The layer parameters (weights and biases) are updated accordingly using the computed gradient not by the layer itself, instead they are updated by the network that holds the instantiated layer.

Parameters
*(input) The input parameter used for calculating the gradient.
*(error) The calculated error.
*(gradient) The calculated gradient.

Reimplemented in ConvolutionType< ForwardConvolutionRule, BackwardConvolutionRule, GradientConvolutionRule, MatType >, MultiLayer< MatType >, MultiLayer< CustomLayers... >, MultiLayer< InputType, OutputType >, LSTMType< MatType >, LinearType< MatType, RegularizerType >, ConcatType< MatType >, DropConnectType< MatType >, Linear3DType< MatType, RegularizerType >, LinearNoBiasType< MatType, RegularizerType >, NoisyLinearType< MatType >, and AddType< MatType >.

Definition at line 173 of file layer.hpp.

Referenced by RecurrentAttention< InputType, OutputType >::Rho().

◆ InputDimensions() [1/2]

const std::vector<size_t>& InputDimensions ( ) const
inline

Get the input dimensions.

Definition at line 222 of file layer.hpp.

Referenced by WeightNormType< InputType, OutputType >::OutputDimensions().

◆ InputDimensions() [2/2]

std::vector<size_t>& InputDimensions ( )
inline

Modify the input dimensions.

Definition at line 224 of file layer.hpp.

◆ Loss()

virtual double Loss ( )
inlinevirtual

Get the layer loss.

Overload this if the layer should add any extra loss to the loss function when computing the objective. (TODO: better comment)

Reimplemented in ReparametrizationType< InputType, OutputType >.

Definition at line 219 of file layer.hpp.

◆ operator=() [1/2]

virtual Layer& operator= ( const Layer< MatType > &  layer)
inlinevirtual

Copy assignment operator. This is not responsible for copying weights!

Definition at line 82 of file layer.hpp.

◆ operator=() [2/2]

virtual Layer& operator= ( Layer< MatType > &&  layer)
inlinevirtual

Move assignment operator. This is not responsible for moving weights!

Definition at line 96 of file layer.hpp.

◆ OutputDimensions()

const std::vector<size_t>& OutputDimensions ( )
inline

Get the output dimensions.

Definition at line 231 of file layer.hpp.

Referenced by ConcatType< MatType >::ComputeOutputDimensions(), and WeightNormType< InputType, OutputType >::OutputDimensions().

◆ OutputSize()

virtual size_t OutputSize ( )
inlinefinalvirtual

Get the number of elements in the output from this layer.

This cannot be overloaded! Overload ComputeOutputDimensions() instead.

Definition at line 267 of file layer.hpp.

◆ Parameters() [1/2]

◆ Parameters() [2/2]

◆ serialize()

void serialize ( Archive &  ar,
const uint32_t   
)
inline

Serialize the layer.

Definition at line 283 of file layer.hpp.

◆ SetWeights()

virtual void SetWeights ( typename MatType::elem_type *  )
inlinevirtual

Reset the layer parameter.

The method is called to assigned the allocated memory to the internal layer parameters like weights and biases. The method should be called before the first call of Forward(input, output). If you do not respect this rule, Forward(input, output) and Backward(input, gy, g) might compute incorrect results.

Parameters
weightsPtrThis pointer should be used as the first element of the memory that is allocated for this layer. In general, SetWeights() implementations should use MakeAlias() with weightsPtr to wrap the weights of a layer.

Reimplemented in ConvolutionType< ForwardConvolutionRule, BackwardConvolutionRule, GradientConvolutionRule, MatType >, MultiLayer< MatType >, MultiLayer< CustomLayers... >, MultiLayer< InputType, OutputType >, DropConnectType< MatType >, AddType< MatType >, LSTMType< MatType >, LinearType< MatType, RegularizerType >, Linear3DType< MatType, RegularizerType >, LinearNoBiasType< MatType, RegularizerType >, and NoisyLinearType< MatType >.

Definition at line 190 of file layer.hpp.

◆ Training() [1/2]

virtual bool const& Training ( ) const
inlinevirtual

Get whether the layer is currently in training mode.

Note
During network training, this should be set to true for each layer in the network, and when predicting/testing the network, this should be set to false. (This is handled automatically by the FFN class and other related classes.)

Definition at line 205 of file layer.hpp.

◆ Training() [2/2]

virtual bool& Training ( )
inlinevirtual

Modify whether the layer is currently in training mode.

Note
During network training, this should be set to true for each layer in the network, and when predicting/testing the network, this should be set to false. (This is handled automatically by the FFN class and other related classes.)

Definition at line 215 of file layer.hpp.

◆ WeightSize()

virtual size_t WeightSize ( ) const
inlinevirtual

Get the total number of trainable weights in the layer.

Reimplemented in TransposedConvolutionType< ForwardConvolutionRule, BackwardConvolutionRule, GradientConvolutionRule, InputType, OutputType >, AtrousConvolution< ForwardConvolutionRule, BackwardConvolutionRule, GradientConvolutionRule, InputType, OutputType >, ConvolutionType< ForwardConvolutionRule, BackwardConvolutionRule, GradientConvolutionRule, MatType >, MultiheadAttentionType< InputType, OutputType, RegularizerType >, FastLSTMType< InputType, OutputType >, BatchNorm< InputType, OutputType >, BatchNorm< InputDataType, OutputDataType >, LSTMType< MatType >, MultiLayer< MatType >, MultiLayer< CustomLayers... >, MultiLayer< InputType, OutputType >, LinearType< MatType, RegularizerType >, LayerNormType< InputType, OutputType >, MultiheadAttentionType< InputType, OutputType, RegularizerType >, WeightNormType< InputType, OutputType >, Linear3DType< MatType, RegularizerType >, VirtualBatchNormType< InputType, OutputType >, DropConnectType< MatType >, FlexibleReLUType< InputType, OutputType >, HighwayType< InputType, OutputType >, LinearNoBiasType< MatType, RegularizerType >, MiniBatchDiscrimination< InputType, OutputType >, MultiplyMergeType< InputType, OutputType >, NoisyLinearType< MatType >, PReLUType< InputType, OutputType >, RBFType< MatType, Activation >, LookupType< InputType, OutputType >, and AddType< MatType >.

Definition at line 195 of file layer.hpp.

Referenced by WeightNormType< InputType, OutputType >::WeightSize().

Member Data Documentation

◆ inputDimensions

std::vector<size_t> inputDimensions
protected

Logical input dimensions of each point.

Although each point given to ! Forward() will be represented as a column in a matrix, logically speaking it can be a higher-order tensor. So, for instance, if the point is 2-dimensional images of size 10x10, Forward() will contain columns with 100 rows, and inputDimensions will be {10, 10}. This generalizes to higher dimensions.

Definition at line 302 of file layer.hpp.

Referenced by ConcatType< MatType >::ComputeOutputDimensions(), LSTMType< MatType >::ComputeOutputDimensions(), Layer< InputDataType, OutputDataType >::ComputeOutputDimensions(), Layer< InputDataType, OutputDataType >::InputDimensions(), Layer< InputDataType, OutputDataType >::operator=(), Layer< InputDataType, OutputDataType >::serialize(), and Linear3DType< MatType, RegularizerType >::WeightSize().

◆ outputDimensions

std::vector<size_t> outputDimensions
protected

Logical output dimensions of each point.

If the layer only performs elementwise operations, this is most likely equal to inputDimensions; but if the layer performs more complicated transformations, it may be different.

Definition at line 310 of file layer.hpp.

Referenced by ConcatType< MatType >::ComputeOutputDimensions(), LSTMType< MatType >::ComputeOutputDimensions(), Layer< InputDataType, OutputDataType >::ComputeOutputDimensions(), Layer< InputDataType, OutputDataType >::operator=(), Layer< InputDataType, OutputDataType >::OutputDimensions(), Layer< InputDataType, OutputDataType >::OutputSize(), and Layer< InputDataType, OutputDataType >::serialize().

◆ training

bool training
protected

If true, the layer is in training mode; otherwise, it is in testing mode.

Definition at line 317 of file layer.hpp.

Referenced by Layer< InputDataType, OutputDataType >::operator=(), Layer< InputDataType, OutputDataType >::serialize(), and Layer< InputDataType, OutputDataType >::Training().

◆ validOutputDimensions


The documentation for this class was generated from the following file:
  • /home/jenkins-mlpack/mlpack.org/_src/mlpack-git/src/mlpack/methods/ann/layer/layer.hpp