FastLSTMType< InputType, OutputType > Class Template Reference

An implementation of a faster version of the Fast LSTM network layer. More...

Inheritance diagram for FastLSTMType< InputType, OutputType >:

Public Types

typedef InputType::elem_type InputET
 
typedef OutputType::elem_type OutputET
 

Public Member Functions

 FastLSTMType ()
 Create the FastLSTMType object. More...

 
 FastLSTMType (const FastLSTMType &layer)
 Copy Constructor. More...

 
 FastLSTMType (FastLSTMType &&layer)
 Move Constructor. More...

 
 FastLSTMType (const size_t inSize, const size_t outSize, const size_t rho=std::numeric_limits< size_t >::max())
 Create the Fast LSTM layer object using the specified parameters. More...

 
void Backward (const InputType &input, const OutputType &gy, OutputType &g)
 Ordinary feed backward pass of a neural network, calculating the function f(x) by propagating x backwards trough f. More...

 
FastLSTMTypeClone () const
 Clone the FastLSTMType object. This handles polymorphism correctly. More...

 
void Forward (const InputType &input, OutputType &output)
 Ordinary feed forward pass of a neural network, evaluating the function f(x) by propagating the activity forward through f. More...

 
void Gradient (const InputType &input, const OutputType &error, OutputType &gradient)
 Calculate the gradient using the output delta and the input activation. More...

 
size_t InSize () const
 Get the number of input units. More...

 
FastLSTMTypeoperator= (const FastLSTMType &layer)
 Copy assignment operator. More...

 
FastLSTMTypeoperator= (FastLSTMType &&layer)
 Move assignment operator. More...

 
const std::vector< size_t > OutputDimensions () const
 
size_t OutSize () const
 Get the number of output units. More...

 
OutputType const & Parameters () const
 Get the parameters. More...

 
OutputType & Parameters ()
 Modify the parameters. More...

 
void Reset ()
 Reset the layer parameter. More...

 
void ResetCell (const size_t size)
 Resets the cell to accept a new input. More...

 
size_t Rho () const
 Get the maximum number of steps to backpropagate through time (BPTT). More...

 
size_t & Rho ()
 Modify the maximum number of steps to backpropagate through time (BPTT). More...

 
template
<
typename
Archive
>
void serialize (Archive &ar, const uint32_t)
 Serialize the layer. More...

 
size_t WeightSize () const
 Get the size of the weight matrix. More...

 
- Public Member Functions inherited from Layer< InputType, OutputType >
 Layer ()
 Default constructor. More...

 
 Layer (const Layer &layer)
 Copy constructor. This is not responsible for copying weights! More...

 
 Layer (Layer &&layer)
 Move constructor. This is not responsible for moving weights! More...

 
virtual ~Layer ()
 Default deconstructor. More...

 
virtual void Backward (const InputType &, const InputType &, InputType &)
 Performs a backpropagation step through the layer, with respect to the given input. More...

 
virtual void ComputeOutputDimensions ()
 Compute the output dimensions. More...

 
virtual void CustomInitialize (InputType &, const size_t)
 Override the weight matrix of the layer. More...

 
virtual void Forward (const InputType &, InputType &)
 Takes an input object, and computes the corresponding output of the layer. More...

 
virtual void Forward (const InputType &, const InputType &)
 Takes an input and output object, and computes the corresponding loss of the layer. More...

 
virtual void Gradient (const InputType &, const InputType &, InputType &)
 Computing the gradient of the layer with respect to its own input. More...

 
const std::vector< size_t > & InputDimensions () const
 Get the input dimensions. More...

 
std::vector< size_t > & InputDimensions ()
 Modify the input dimensions. More...

 
virtual double Loss ()
 Get the layer loss. More...

 
virtual Layeroperator= (const Layer &layer)
 Copy assignment operator. This is not responsible for copying weights! More...

 
virtual Layeroperator= (Layer &&layer)
 Move assignment operator. This is not responsible for moving weights! More...

 
const std::vector< size_t > & OutputDimensions ()
 Get the output dimensions. More...

 
virtual size_t OutputSize () final
 Get the number of elements in the output from this layer. More...

 
void serialize (Archive &ar, const uint32_t)
 Serialize the layer. More...

 
virtual void SetWeights (typename InputType ::elem_type *)
 Reset the layer parameter. More...

 
virtual bool const & Training () const
 Get whether the layer is currently in training mode. More...

 
virtual bool & Training ()
 Modify whether the layer is currently in training mode. More...

 

Additional Inherited Members

- Protected Attributes inherited from Layer< InputType, OutputType >
std::vector< size_t > inputDimensions
 Logical input dimensions of each point. More...

 
std::vector< size_t > outputDimensions
 Logical output dimensions of each point. More...

 
bool training
 If true, the layer is in training mode; otherwise, it is in testing mode. More...

 
bool validOutputDimensions
 This is true if ComputeOutputDimensions() has been called, and outputDimensions can be considered to be up-to-date. More...

 

Detailed Description


template
<
typename
InputType
=
arma::mat
,
typename
OutputType
=
arma::mat
>

class mlpack::ann::FastLSTMType< InputType, OutputType >

An implementation of a faster version of the Fast LSTM network layer.

Basically by combining the calculation of the input, forget, output gates and hidden state in a single step. The standard formula changes as follows:

\begin{eqnarray} i &=& sigmoid(W \cdot x + W \cdot h + b) \\ f &=& sigmoid(W \cdot x + W \cdot h + b) \\ z &=& tanh(W \cdot x + W \cdot h + b) \\ c &=& f \cdot c + i \cdot z \\ o &=& sigmoid(W \cdot x + W \cdot h + b) \\ h &=& o \cdot tanh(c) \end{eqnarray}

Note that FastLSTM network layer does not use peephole connections between the cell and gates.

Note also that if a FastLSTM layer is desired as the first layer of a neural network, an IdentityLayer should be added to the network as the first layer, and then the FastLSTM layer should be added.

For more information, see the following.

@article{Hochreiter1997,
author = {Hochreiter, Sepp and Schmidhuber, J\"{u}rgen},
title = {Long Short-term Memory},
journal = {Neural Comput.},
year = {1997},
url = {https://www.bioinf.jku.at/publications/older/2604.pdf}
}
See also
LSTM for a standard implementation of the LSTM layer.
Template Parameters
InputTypeType of the input data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
OutputTypeType of the output data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).

Definition at line 68 of file fast_lstm.hpp.

Member Typedef Documentation

◆ InputET

typedef InputType::elem_type InputET

Definition at line 72 of file fast_lstm.hpp.

◆ OutputET

typedef OutputType::elem_type OutputET

Definition at line 73 of file fast_lstm.hpp.

Constructor & Destructor Documentation

◆ FastLSTMType() [1/4]

◆ FastLSTMType() [2/4]

FastLSTMType ( const FastLSTMType< InputType, OutputType > &  layer)

Copy Constructor.

◆ FastLSTMType() [3/4]

FastLSTMType ( FastLSTMType< InputType, OutputType > &&  layer)

Move Constructor.

◆ FastLSTMType() [4/4]

FastLSTMType ( const size_t  inSize,
const size_t  outSize,
const size_t  rho = std::numeric_limits< size_t >::max() 
)

Create the Fast LSTM layer object using the specified parameters.

Parameters
inSizeThe number of input units.
outSizeThe number of output units.
rhoMaximum number of steps to backpropagate through time (BPTT).

Member Function Documentation

◆ Backward()

void Backward ( const InputType &  input,
const OutputType &  gy,
OutputType &  g 
)

Ordinary feed backward pass of a neural network, calculating the function f(x) by propagating x backwards trough f.

Using the results from the feed forward pass.

Parameters
inputThe propagated input activation.
gyThe backpropagated error.
gThe calculated gradient.

Referenced by FastLSTMType< InputType, OutputType >::Clone().

◆ Clone()

◆ Forward()

void Forward ( const InputType &  input,
OutputType &  output 
)

Ordinary feed forward pass of a neural network, evaluating the function f(x) by propagating the activity forward through f.

Parameters
inputInput data used for evaluating the specified function.
outputResulting output activation.

Referenced by FastLSTMType< InputType, OutputType >::Clone().

◆ Gradient()

void Gradient ( const InputType &  input,
const OutputType &  error,
OutputType &  gradient 
)

Calculate the gradient using the output delta and the input activation.

Parameters
inputThe input parameter used for calculating the gradient.
errorThe calculated error.
gradientThe calculated gradient.

Referenced by FastLSTMType< InputType, OutputType >::Clone().

◆ InSize()

size_t InSize ( ) const
inline

Get the number of input units.

Definition at line 161 of file fast_lstm.hpp.

◆ operator=() [1/2]

FastLSTMType& operator= ( const FastLSTMType< InputType, OutputType > &  layer)

Copy assignment operator.

◆ operator=() [2/2]

FastLSTMType& operator= ( FastLSTMType< InputType, OutputType > &&  layer)

Move assignment operator.

◆ OutputDimensions()

const std::vector<size_t> OutputDimensions ( ) const
inline

◆ OutSize()

size_t OutSize ( ) const
inline

Get the number of output units.

Definition at line 164 of file fast_lstm.hpp.

◆ Parameters() [1/2]

OutputType const& Parameters ( ) const
inlinevirtual

Get the parameters.

Reimplemented from Layer< InputType, OutputType >.

Definition at line 156 of file fast_lstm.hpp.

◆ Parameters() [2/2]

OutputType& Parameters ( )
inlinevirtual

Modify the parameters.

Reimplemented from Layer< InputType, OutputType >.

Definition at line 158 of file fast_lstm.hpp.

◆ Reset()

void Reset ( )

Reset the layer parameter.

Referenced by FastLSTMType< InputType, OutputType >::Clone().

◆ ResetCell()

void ResetCell ( const size_t  size)

Resets the cell to accept a new input.

This breaks the BPTT chain starts a new one.

Parameters
sizeThe current maximum number of steps through time.

Referenced by FastLSTMType< InputType, OutputType >::Clone().

◆ Rho() [1/2]

size_t Rho ( ) const
inline

Get the maximum number of steps to backpropagate through time (BPTT).

Definition at line 151 of file fast_lstm.hpp.

◆ Rho() [2/2]

size_t& Rho ( )
inline

Modify the maximum number of steps to backpropagate through time (BPTT).

Definition at line 153 of file fast_lstm.hpp.

◆ serialize()

void serialize ( Archive &  ar,
const uint32_t   
)

Serialize the layer.

Referenced by FastLSTMType< InputType, OutputType >::OutputDimensions().

◆ WeightSize()

size_t WeightSize ( ) const
inlinevirtual

Get the size of the weight matrix.

Reimplemented from Layer< InputType, OutputType >.

Definition at line 167 of file fast_lstm.hpp.


The documentation for this class was generated from the following file:
  • /home/jenkins-mlpack/mlpack.org/_src/mlpack-git/src/mlpack/methods/ann/layer/not_adapted/fast_lstm.hpp