FlexibleReLU< InputDataType, OutputDataType > Class Template Reference

The FlexibleReLU activation function, defined by. More...

Public Member Functions

 FlexibleReLU (const double alpha=0)
 Create the FlexibleReLU object using the specified parameters. More...

 
double const & Alpha () const
 Get the parameter controlling the range of the relu function. More...

 
double & Alpha ()
 Modify the parameter controlling the range of the relu function. More...

 
template
<
typename
DataType
>
void Backward (const DataType &&input, DataType &&gy, DataType &&g)
 Ordinary feed backward pass of a neural network, calculating the function f(x) by propagating x backwards through f. More...

 
OutputDataType const & Delta () const
 Get the delta. More...

 
OutputDataType & Delta ()
 Modify the delta. More...

 
template
<
typename
InputType
,
typename
OutputType
>
void Forward (const InputType &&input, OutputType &&output)
 Ordinary feed forward pass of a neural network, evaluating the function f(x) by propagating the activity forward through f. More...

 
template
<
typename
eT
>
void Gradient (const arma::Mat< eT > &&input, arma::Mat< eT > &&error, arma::Mat< eT > &&gradient)
 Calculate the gradient using the output delta and the input activation. More...

 
OutputDataType const & Gradient () const
 Get the gradient. More...

 
OutputDataType & Gradient ()
 Modify the gradient. More...

 
OutputDataType const & OutputParameter () const
 Get the output parameter. More...

 
OutputDataType & OutputParameter ()
 Modify the output parameter. More...

 
OutputDataType const & Parameters () const
 Get the parameters. More...

 
OutputDataType & Parameters ()
 Modify the parameters. More...

 
void Reset ()
 Reset the layer parameter. More...

 
template
<
typename
Archive
>
void serialize (Archive &ar, const unsigned int)
 Serialize the layer. More...

 

Detailed Description


template
<
typename
InputDataType
=
arma::mat
,
typename
OutputDataType
=
arma::mat
>

class mlpack::ann::FlexibleReLU< InputDataType, OutputDataType >

The FlexibleReLU activation function, defined by.

\begin{eqnarray*} f(x) &=& \max(0,x)+alpha \\ f'(x) &=& \left\{ \begin{array}{lr} 1 & : x > 0 \\ 0 & : x \le 0 \end{array} \right. \end{eqnarray*}

For more information, read the following paper:

@article{Qiu2018,
author = {Suo Qiu, Xiangmin Xu and Bolun Cai},
title = {FReLU: Flexible Rectified Linear Units for Improving
Convolutional Neural Networks}
journal = {arxiv preprint},
URL = {https://arxiv.org/abs/1706.08098},
year = {2018}
}
Template Parameters
InputDataTypeType of the input data (arma::colvec, arma::mar, arma::sp_mat or arma::cube)
OutputDataTypeType of the output data (arma::colvec, arma::mat, arma::sp_mat or arma::cube)

Definition at line 59 of file flexible_relu.hpp.

Constructor & Destructor Documentation

◆ FlexibleReLU()

FlexibleReLU ( const double  alpha = 0)

Create the FlexibleReLU object using the specified parameters.

The non zero parameter can be adjusted by specifying the parameter alpha which controls the range of the relu function. (Default alpha = 0) This parameter is trainable.

Parameters
alphaParameter for adjusting the range of the relu function.

Member Function Documentation

◆ Alpha() [1/2]

double const& Alpha ( ) const
inline

Get the parameter controlling the range of the relu function.

Definition at line 134 of file flexible_relu.hpp.

◆ Alpha() [2/2]

double& Alpha ( )
inline

Modify the parameter controlling the range of the relu function.

Definition at line 136 of file flexible_relu.hpp.

References FlexibleReLU< InputDataType, OutputDataType >::serialize().

◆ Backward()

void Backward ( const DataType &&  input,
DataType &&  gy,
DataType &&  g 
)

Ordinary feed backward pass of a neural network, calculating the function f(x) by propagating x backwards through f.

Using the results from the feed forward pass.

Parameters
inputThe propagated input activation.
gyThe backpropagated error.
gThe calculated gradient.

◆ Delta() [1/2]

OutputDataType const& Delta ( ) const
inline

Get the delta.

Definition at line 124 of file flexible_relu.hpp.

◆ Delta() [2/2]

OutputDataType& Delta ( )
inline

Modify the delta.

Definition at line 126 of file flexible_relu.hpp.

◆ Forward()

void Forward ( const InputType &&  input,
OutputType &&  output 
)

Ordinary feed forward pass of a neural network, evaluating the function f(x) by propagating the activity forward through f.

Parameters
inputInput data used for evaluating the specified function.
outputResulting output activation.

◆ Gradient() [1/3]

void Gradient ( const arma::Mat< eT > &&  input,
arma::Mat< eT > &&  error,
arma::Mat< eT > &&  gradient 
)

Calculate the gradient using the output delta and the input activation.

Parameters
inputThe input parameter used for calculating the gradient.
errorThe calculated error.
gradientThe calculated gradient.

◆ Gradient() [2/3]

OutputDataType const& Gradient ( ) const
inline

Get the gradient.

Definition at line 129 of file flexible_relu.hpp.

◆ Gradient() [3/3]

OutputDataType& Gradient ( )
inline

Modify the gradient.

Definition at line 131 of file flexible_relu.hpp.

◆ OutputParameter() [1/2]

OutputDataType const& OutputParameter ( ) const
inline

Get the output parameter.

Definition at line 119 of file flexible_relu.hpp.

◆ OutputParameter() [2/2]

OutputDataType& OutputParameter ( )
inline

Modify the output parameter.

Definition at line 121 of file flexible_relu.hpp.

◆ Parameters() [1/2]

OutputDataType const& Parameters ( ) const
inline

Get the parameters.

Definition at line 114 of file flexible_relu.hpp.

◆ Parameters() [2/2]

OutputDataType& Parameters ( )
inline

Modify the parameters.

Definition at line 116 of file flexible_relu.hpp.

◆ Reset()

void Reset ( )

Reset the layer parameter.

◆ serialize()

void serialize ( Archive &  ar,
const unsigned  int 
)

Serialize the layer.

Referenced by FlexibleReLU< InputDataType, OutputDataType >::Alpha().


The documentation for this class was generated from the following file:
  • /home/jenkins-mlpack/mlpack.org/_src/mlpack-3.2.1/src/mlpack/methods/ann/layer/flexible_relu.hpp