Implementation of the Highway layer. More...

Inheritance diagram for HighwayType< InputType, OutputType >:

[legend]

Public Member Functions
	HighwayType ()
	Create the HighwayType object. More...

	HighwayType (const HighwayType &other)
	Copy the given HighwayType (but not weights). More...

	HighwayType (HighwayType &&other)
	Take ownership of the given HighwayType (but not weights). More...

virtual	~HighwayType ()
	Destroy the HighwayType object. More...

void	Backward (const InputType &, const OutputType &gy, OutputType &g)
	Ordinary feed-backward pass of a neural network, calculating the function f(x) by propagating x backwards through f. More...

HighwayType *	Clone () const
	Clone the HighwayType object. This handles polymorphism correctly. More...

void	Forward (const InputType &input, OutputType &output)
	Ordinary feed-forward pass of a neural network, evaluating the function f(x) by propagating the activity forward through f. More...

void	Gradient (const InputType &input, const OutputType &error, OutputType &gradient)
	Calculate the gradient using the output delta and the input activation. More...

HighwayType &	operator= (const HighwayType &other)
	Copy the given HighwayType (but not weights). More...

HighwayType &	operator= (HighwayType &&other)
	Take ownership of the given HighwayType (but not weights). More...

OutputType const &	Parameters () const
	Get the parameters. More...

OutputType &	Parameters ()
	Modify the parameters. More...

template < typename Archive >
void	serialize (Archive &ar, const uint32_t)
	Serialize the layer. More...

void	SetWeights (typename OutputType::elem_type *weightsPtr)

size_t	WeightSize () const
	Get the number of trainable weights. More...

Public Member Functions inherited from MultiLayer< InputType, OutputType >
	MultiLayer ()
	Create an empty MultiLayer that holds no layers of its own. More...

	MultiLayer (const MultiLayer &other)
	Copy the given MultiLayer. More...

	MultiLayer (MultiLayer &&other)
	Take ownership of the layers of the given MultiLayer. More...

virtual	~MultiLayer ()
	Virtual destructor: delete all held layers. More...

void	Add (Args... args)

void	Add (Layer< InputType > *layer)

virtual void	Backward (const InputType &input, const InputType &gy, InputType &g)
	Perform a backward pass with the given data. More...

virtual void	ComputeOutputDimensions ()
	Compute the output dimensions of the MultiLayer using `InputDimensions()`. More...

virtual void	Forward (const InputType &input, InputType &output)
	Perform a forward pass with the given input data. More...

void	Forward (const InputType &input, InputType &output, const size_t start, const size_t end)
	Perform a forward pass with the given input data, but only on a subset of the layers in the MultiLayer. More...

virtual void	Gradient (const InputType &input, const InputType &error, InputType &gradient)
	Compute the gradients of each layer. More...

virtual double	Loss () const
	Compute the loss that should be added to the objective. More...

const std::vector< Layer< InputType > *>	Network () const
	Get the network (series of layers) held by this MultiLayer. More...

std::vector< Layer< InputType > *> &	Network ()
	Modify the network (series of layers) held by this MultiLayer. More...

MultiLayer &	operator= (const MultiLayer &other)
	Copy the given MultiLayer. More...

MultiLayer &	operator= (MultiLayer &&other)
	Take ownership of the given MultiLayer. More...

void	serialize (Archive &ar, const uint32_t)
	Serialize the MultiLayer. More...

virtual void	SetWeights (typename InputType ::elem_type *weightsPtr)
	Set the weights of the layer to use the memory given as `weightsPtr`. More...

Public Member Functions inherited from Layer< InputType >
	Layer ()
	Default constructor. More...

	Layer (const Layer &layer)
	Copy constructor. This is not responsible for copying weights! More...

	Layer (Layer &&layer)
	Move constructor. This is not responsible for moving weights! More...

virtual	~Layer ()
	Default deconstructor. More...

virtual void	Forward (const InputType &, const InputType &)
	Takes an input and output object, and computes the corresponding loss of the layer. More...

const std::vector< size_t > &	InputDimensions () const
	Get the input dimensions. More...

std::vector< size_t > &	InputDimensions ()
	Modify the input dimensions. More...

virtual double	Loss ()
	Get the layer loss. More...

virtual Layer &	operator= (const Layer &layer)
	Copy assignment operator. This is not responsible for copying weights! More...

virtual Layer &	operator= (Layer &&layer)
	Move assignment operator. This is not responsible for moving weights! More...

const std::vector< size_t > &	OutputDimensions ()
	Get the output dimensions. More...

virtual size_t	OutputSize () final
	Get the number of elements in the output from this layer. More...

void	serialize (Archive &ar, const uint32_t)
	Serialize the layer. More...

virtual bool const &	Training () const
	Get whether the layer is currently in training mode. More...

virtual bool &	Training ()
	Modify whether the layer is currently in training mode. More...

Additional Inherited Members
Protected Member Functions inherited from MultiLayer< InputType, OutputType >
void	InitializeBackwardPassMemory (const size_t batchSize)
	Initialize memory that will be used by each layer for the backwards pass, assuming that the input will have the given `batchSize`. More...

void	InitializeForwardPassMemory (const size_t batchSize)
	Initialize memory that will be used by each layer for the forward pass, assuming that the input will have the given `batchSize`. More...

void	InitializeGradientPassMemory (InputType &gradient)
	Initialize memory for the gradient pass. More...

Protected Attributes inherited from MultiLayer< InputType, OutputType >
size_t	inSize

InputType	layerDeltaMatrix
	This matrix stores all of the backwards pass results of each layer when Backward() is called. More...

std::vector< InputType >	layerDeltas
	These are aliases of `layerDeltaMatrix` for each layer. More...

std::vector< InputType >	layerGradients
	Gradient aliases for each layer. More...

InputType	layerOutputMatrix
	This matrix stores all of the outputs of each layer when Forward() is called. More...

std::vector< InputType >	layerOutputs
	These are aliases of `layerOutputMatrix` for each layer. More...

std::vector< Layer< InputType > *>	network
	The internally-held network. More...

size_t	totalInputSize

size_t	totalOutputSize

Protected Attributes inherited from Layer< InputType >
std::vector< size_t >	inputDimensions
	Logical input dimensions of each point. More...

std::vector< size_t >	outputDimensions
	Logical output dimensions of each point. More...

bool	training
	If true, the layer is in training mode; otherwise, it is in testing mode. More...

bool	validOutputDimensions
	This is `true` if `ComputeOutputDimensions()` has been called, and `outputDimensions` can be considered to be up-to-date. More...

Detailed Description

template
<
typename
InputType
=
arma::mat
,
typename
OutputType
=
arma::mat
>

class mlpack::ann::HighwayType< InputType, OutputType >

Implementation of the Highway layer.

The Highway class can vary its behavior between that of feed-forward fully connected network container and that of a layer which simply passes its inputs through depending on the transform gate. Note that the size of the input and output matrices of this class should be equal.

For more information, refer the following paper.

@article{Srivastava2015,
  author  = {Rupesh Kumar Srivastava, Klaus Greff, Jurgen Schmidhuber},
  title   = {Training Very Deep Networks},
  journal = {Advances in Neural Information Processing Systems},
  year    = {2015},
  url     = {https://arxiv.org/abs/1507.06228},
}

Template Parameters

InputType	Type of the input data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).
OutputType	Type of the output data (arma::colvec, arma::mat, arma::sp_mat or arma::cube).

Definition at line 52 of file highway.hpp.

Constructor & Destructor Documentation

◆ HighwayType() [1/3]

HighwayType ( )

Create the HighwayType object.

Referenced by HighwayType< InputType, OutputType >::Clone().

◆ ~HighwayType()

virtual ~HighwayType ( )

virtual

Destroy the HighwayType object.

◆ HighwayType() [2/3]

HighwayType ( const HighwayType< InputType, OutputType > & other )

Copy the given HighwayType (but not weights).

◆ HighwayType() [3/3]

HighwayType ( HighwayType< InputType, OutputType > && other )

Take ownership of the given HighwayType (but not weights).

Member Function Documentation

◆ Backward()

void Backward	(	const InputType &	,
		const OutputType &	gy,
		OutputType &	g
	)

Ordinary feed-backward pass of a neural network, calculating the function f(x) by propagating x backwards through f.

Using the results from the feed-forward pass.

Parameters

*	(input) The propagated input activation.
gy	The backpropagated error.
g	The calculated gradient.

Referenced by HighwayType< InputType, OutputType >::Clone().

◆ Clone()

HighwayType* Clone ( ) const

inlinevirtual

Clone the HighwayType object. This handles polymorphism correctly.

Reimplemented from MultiLayer< InputType, OutputType >.

Definition at line 62 of file highway.hpp.

References HighwayType< InputType, OutputType >::Backward(), HighwayType< InputType, OutputType >::Forward(), HighwayType< InputType, OutputType >::Gradient(), HighwayType< InputType, OutputType >::HighwayType(), HighwayType< InputType, OutputType >::operator=(), and HighwayType< InputType, OutputType >::SetWeights().

◆ Forward()

void Forward	(	const InputType &	input,
		OutputType &	output
	)

Ordinary feed-forward pass of a neural network, evaluating the function f(x) by propagating the activity forward through f.

Parameters

input	Input data used for evaluating the specified function.
output	Resulting output activation.

Referenced by HighwayType< InputType, OutputType >::Clone().

◆ Gradient()

void Gradient	(	const InputType &	input,
		const OutputType &	error,
		OutputType &	gradient
	)

Calculate the gradient using the output delta and the input activation.

Parameters

input	The input parameter used for calculating the gradient.
error	The calculated error.
gradient	The calculated gradient.

Referenced by HighwayType< InputType, OutputType >::Clone().

◆ operator=() [1/2]

HighwayType& operator= ( const HighwayType< InputType, OutputType > & other )

Copy the given HighwayType (but not weights).

Referenced by HighwayType< InputType, OutputType >::Clone().

◆ operator=() [2/2]

HighwayType& operator= ( HighwayType< InputType, OutputType > && other )

Take ownership of the given HighwayType (but not weights).

◆ Parameters() [1/2]

OutputType const& Parameters ( ) const

inlinevirtual

Get the parameters.

Reimplemented from Layer< InputType >.

Definition at line 109 of file highway.hpp.

◆ Parameters() [2/2]

OutputType& Parameters ( )

inlinevirtual

Modify the parameters.

Reimplemented from Layer< InputType >.

Definition at line 111 of file highway.hpp.

◆ serialize()

void serialize	(	Archive &	ar,
		const uint32_t
	)

Serialize the layer.

Referenced by HighwayType< InputType, OutputType >::WeightSize().

◆ SetWeights()

void SetWeights ( typename OutputType::elem_type * weightsPtr )

Referenced by HighwayType< InputType, OutputType >::Clone().

◆ WeightSize()

size_t WeightSize ( ) const

inlinevirtual

Get the number of trainable weights.

Reimplemented from MultiLayer< InputType, OutputType >.

Definition at line 114 of file highway.hpp.

References MultiLayer< InputType, OutputType >::network, HighwayType< InputType, OutputType >::serialize(), and MultiLayer< InputType, OutputType >::totalInputSize.

The documentation for this class was generated from the following file:

/home/jenkins-mlpack/mlpack.org/_src/mlpack-git/src/mlpack/methods/ann/layer/not_adapted/highway.hpp

Public Member Functions

Additional Inherited Members

Detailed Description

template<typenameInputType=arma::mat,typenameOutputType=arma::mat> class mlpack::ann::HighwayType< InputType, OutputType >

Constructor & Destructor Documentation

◆ HighwayType() [1/3]

◆ ~HighwayType()

◆ HighwayType() [2/3]

◆ HighwayType() [3/3]

Member Function Documentation

◆ Backward()

◆ Clone()

◆ Forward()

◆ Gradient()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ Parameters() [1/2]

◆ Parameters() [2/2]

◆ serialize()

◆ SetWeights()

◆ WeightSize()

template
<
typename
InputType
=
arma::mat
,
typename
OutputType
=
arma::mat
>

class mlpack::ann::HighwayType< InputType, OutputType >