Definition of a standard recurrent neural network container. More...
Public Member Functions | |
RNN (const size_t bpttSteps=0, const bool single=false, OutputLayerType outputLayer=OutputLayerType(), InitializationRuleType initializeRule=InitializationRuleType()) | |
Create the RNN object. More... | |
RNN (const RNN &) | |
Copy constructor. More... | |
RNN (RNN &&) | |
Move constructor. More... | |
~RNN () | |
Destroy the RNN and release any memory it is holding. More... | |
template<typename LayerType , typename... Args> | |
void | Add (Args... args) |
Add a new module to the model. More... | |
void | Add (Layer< MatType > *layer) |
Add a new module to the model. More... | |
size_t | BPTTSteps () const |
Return the number of steps allowed for BPTT. More... | |
size_t & | BPTTSteps () |
Modify the number of steps allowed for BPTT. More... | |
MatType::elem_type | Evaluate (const arma::Cube< typename MatType::elem_type > &predictors, const arma::Cube< typename MatType::elem_type > &responses) |
Evaluate the recurrent network with the given predictors and responses. More... | |
MatType::elem_type | Evaluate (const MatType ¶meters) |
Evaluate the recurrent network with the given parameters. More... | |
MatType::elem_type | Evaluate (const MatType ¶meters, const size_t begin, const size_t batchSize) |
Evaluate the recurrent network with the given parameters, but using only a number of data points. More... | |
template < typename GradType > | |
MatType::elem_type | EvaluateWithGradient (const MatType ¶meters, GradType &gradient) |
Evaluate the recurrent network with the given parameters. More... | |
template < typename GradType > | |
MatType::elem_type | EvaluateWithGradient (const MatType ¶meters, const size_t begin, GradType &gradient, const size_t batchSize) |
Evaluate the recurrent network with the given parameters, but using only a number of data points. More... | |
template < typename GradType > | |
void | Gradient (const MatType ¶meters, const size_t begin, GradType &gradient, const size_t batchSize) |
Evaluate the gradient of the recurrent network with the given parameters, and with respect to only a number of points in the dataset. More... | |
std::vector< size_t > & | InputDimensions () |
Set the logical dimensions of the input. More... | |
const std::vector< size_t > & | InputDimensions () const |
Get the logical dimensions of the input. More... | |
const std::vector< Layer< MatType > * > & | Network () const |
Get the network model. More... | |
std::vector< Layer< MatType > * > & | Network () |
Modify the network model. More... | |
size_t | NumFunctions () const |
Return the number of separable functions (the number of predictor points). More... | |
RNN & | operator= (const RNN &) |
Copy operator. More... | |
RNN & | operator= (RNN &&) |
Move assignment operator. More... | |
const MatType & | Parameters () const |
Return the initial point for the optimization. More... | |
MatType & | Parameters () |
Modify the initial point for the optimization. More... | |
void | Predict (arma::Cube< typename MatType::elem_type > predictors, arma::Cube< typename MatType::elem_type > &results, const size_t batchSize=128) |
Predict the responses to a given set of predictors. More... | |
void | Reset (const size_t inputDimensionality=0) |
Reset the stored data of the network entirely. More... | |
void | ResetData (arma::Cube< typename MatType::elem_type > predictors, arma::Cube< typename MatType::elem_type > responses) |
Prepare the network for the given data. More... | |
template < typename Archive > | |
void | serialize (Archive &ar, const uint32_t) |
Serialize the model. More... | |
void | SetNetworkMode (const bool training) |
Set all the layers in the network to training mode, if training is true , or set all the layers in the network to testing mode, if training is false . More... | |
void | Shuffle () |
Note: this function is implement so that it can be used by ensmallen's optimizers. More... | |
template<typename OptimizerType , typename... CallbackTypes> | |
MatType::elem_type | Train (arma::Cube< typename MatType::elem_type > predictors, arma::Cube< typename MatType::elem_type > responses, OptimizerType &optimizer, CallbackTypes &&... callbacks) |
Train the recurrent network on the given input data using the given optimizer. More... | |
template<typename OptimizerType = ens::RMSProp, typename... CallbackTypes> | |
MatType::elem_type | Train (arma::Cube< typename MatType::elem_type > predictors, arma::Cube< typename MatType::elem_type > responses, CallbackTypes &&... callbacks) |
Train the recurrent network on the given input data. More... | |
size_t | WeightSize () |
Definition of a standard recurrent neural network container.
A recurrent neural network can handle recurrent layers (i.e. RecurrentLayer
s), which hold internal state and are passed sequences of data as inputs.
As opposed to the standard FFN
, which takes data in a matrix format where each column is a data point, the RNN
takes a cube format where each column is a data point and each slice is a time step.
OutputLayerType | The output layer type used to evaluate the network. |
InitializationRuleType | Rule used to initialize the weight matrix. |
Definition at line 24 of file forward_decls.hpp.
RNN | ( | const size_t | bpttSteps = 0 , |
const bool | single = false , |
||
OutputLayerType | outputLayer = OutputLayerType() , |
||
InitializationRuleType | initializeRule = InitializationRuleType() |
||
) |
Create the RNN object.
Optionally, specify which initialize rule and performance function should be used.
If you want to pass in a parameter and discard the original parameter object, be sure to use std::move to avoid unnecessary copy.
bpttSteps | Number of time steps to use for BPTT (backpropagation through time) when training. |
single | If true, then the network will expect only a single timestep for responses. (That is, every input sequence only has one single output; so, responses.n_slices should be 1 when calling Train() .) |
outputLayer | Output layer used to evaluate the network. |
initializeRule | Optional instantiated InitializationRule object for initializing the network parameter. |
|
inline |
|
inline |
|
inline |
MatType::elem_type Evaluate | ( | const arma::Cube< typename MatType::elem_type > & | predictors, |
const arma::Cube< typename MatType::elem_type > & | responses | ||
) |
Evaluate the recurrent network with the given predictors and responses.
This functions is usually used to monitor progress while training.
predictors | Input variables. |
responses | Target outputs for input variables. |
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::SetNetworkMode().
MatType::elem_type Evaluate | ( | const MatType & | parameters | ) |
Evaluate the recurrent network with the given parameters.
This function is usually called by the optimizer to train the model.
parameters | Matrix model parameters. |
MatType::elem_type Evaluate | ( | const MatType & | parameters, |
const size_t | begin, | ||
const size_t | batchSize | ||
) |
Evaluate the recurrent network with the given parameters, but using only a number of data points.
This is useful for optimizers such as SGD, which require a separable objective function.
Note that the network may return different results depending on the mode it is in (see SetNetworkMode()
).
parameters | Matrix model parameters. |
begin | Index of the starting point to use for objective function evaluation. |
batchSize | Number of points to be passed at a time to use for objective function evaluation. |
MatType::elem_type EvaluateWithGradient | ( | const MatType & | parameters, |
GradType & | gradient | ||
) |
Evaluate the recurrent network with the given parameters.
This function is usually called by the optimizer to train the model. This just calls the overload of EvaluateWithGradient() with batchSize = 1.
parameters | Matrix model parameters. |
gradient | Matrix to output gradient into. |
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::SetNetworkMode().
MatType::elem_type EvaluateWithGradient | ( | const MatType & | parameters, |
const size_t | begin, | ||
GradType & | gradient, | ||
const size_t | batchSize | ||
) |
Evaluate the recurrent network with the given parameters, but using only a number of data points.
This is useful for optimizers such as SGD, which require a separable objective function.
parameters | Matrix model parameters. |
begin | Index of the starting point to use for objective function evaluation. |
gradient | Matrix to output gradient into. |
batchSize | Number of points to be passed at a time to use for objective function evaluation. |
void Gradient | ( | const MatType & | parameters, |
const size_t | begin, | ||
GradType & | gradient, | ||
const size_t | batchSize | ||
) |
Evaluate the gradient of the recurrent network with the given parameters, and with respect to only a number of points in the dataset.
This is useful for optimizers such as SGD, which require a separable objective function.
parameters | Matrix of the model parameters to be optimized. |
begin | Index of the starting point to use for objective function gradient evaluation. |
gradient | Matrix to output gradient into. |
batchSize | Number of points to be processed as a batch for objective function gradient evaluation. |
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::SetNetworkMode().
|
inline |
Set the logical dimensions of the input.
Train()
and Predict()
expect data to be passed such that one point corresponds to one column, but this data is allowed to be an arbitrary higher-order tensor.
So, if the input is meant to be 28x28x3 images, then the input data to Train()
or Predict()
should have 28*28*3 = 2352 rows, and InputDImensions()
should be set to { 28, 28, 3}
. Then, the layers of the network will interpret each input point as a 3-dimensional image instead of a 1-dimensional vector.
If InputDimensions()
is left unset before training, the data will be assumed to be a 1-dimensional vector.
|
inline |
|
inline |
|
inline |
Modify the network model.
Be careful! If you change the structure of the network or parameters for layers, its state may become invalid, and the next time it is used for any operation the parameters will be reset.
Don't add any layers like this; use Add()
instead.
|
inline |
Move assignment operator.
|
inline |
|
inline |
void Predict | ( | arma::Cube< typename MatType::elem_type > | predictors, |
arma::Cube< typename MatType::elem_type > & | results, | ||
const size_t | batchSize = 128 |
||
) |
Predict the responses to a given set of predictors.
The responses will reflect the output of the given output layer as returned by the output layer function.
If you want to pass in a parameter and discard the original parameter object, be sure to use std::move to avoid unnecessary copy.
predictors | Input predictors. |
results | Matrix to put output predictions of responses into. |
batchSize | Batch size to use for prediction. |
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::Network().
void Reset | ( | const size_t | inputDimensionality = 0 | ) |
Reset the stored data of the network entirely.
This reset all weights of each layer using InitializationRuleType
, and prepares the network to accept a (flat 1-d) input size of inputDimensionality
(if passed), or whatever input size has been set with InputDimensions()
.
This also resets the mode of the network to prediction mode (not training mode). See SetNetworkMode()
for more information.
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::BPTTSteps().
void ResetData | ( | arma::Cube< typename MatType::elem_type > | predictors, |
arma::Cube< typename MatType::elem_type > | responses | ||
) |
Prepare the network for the given data.
This function won't actually trigger training process.
predictors | Input data variables. |
responses | Outputs results from input data variables. |
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::NumFunctions().
void serialize | ( | Archive & | ar, |
const uint32_t | |||
) |
Serialize the model.
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::SetNetworkMode().
|
inline |
void Shuffle | ( | ) |
Note: this function is implement so that it can be used by ensmallen's optimizers.
It's not generally meant to be used otherwise.
Shuffle the order of function visitation. (This is equivalent to shuffling the dataset during training.)
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::NumFunctions().
MatType::elem_type Train | ( | arma::Cube< typename MatType::elem_type > | predictors, |
arma::Cube< typename MatType::elem_type > | responses, | ||
OptimizerType & | optimizer, | ||
CallbackTypes &&... | callbacks | ||
) |
Train the recurrent network on the given input data using the given optimizer.
This will use the existing model parameters as a starting point for the optimization. If this is not what you want, then you should access the parameters vector directly with Parameters() and modify it as desired.
If you want to pass in a parameter and discard the original parameter object, be sure to use std::move to avoid unnecessary copy.
OptimizerType | Type of optimizer to use to train the model. |
CallbackTypes | Types of Callback Functions. |
predictors | Input training variables. |
responses | Outputs results from input training variables. |
optimizer | Instantiated optimizer used to train the model. |
callbacks | Callback function for ensmallen optimizer OptimizerType . See https://www.ensmallen.org/docs.html#callback-documentation. |
Referenced by RNN< OutputLayerType, InitializationRuleType, CustomLayers... >::Network().
MatType::elem_type Train | ( | arma::Cube< typename MatType::elem_type > | predictors, |
arma::Cube< typename MatType::elem_type > | responses, | ||
CallbackTypes &&... | callbacks | ||
) |
Train the recurrent network on the given input data.
By default, the RMSProp optimization algorithm is used, but others can be specified (such as ens::SGD).
This will use the existing model parameters as a starting point for the optimization. If this is not what you want, then you should access the parameters vector directly with Parameters() and modify it as desired.
If you want to pass in a parameter and discard the original parameter object, be sure to use std::move to avoid unnecessary copy.
OptimizerType | Type of optimizer to use to train the model. |
predictors | Input training variables. |
CallbackTypes | Types of Callback Functions. |
responses | Outputs results from input training variables. |
callbacks | Callback function for ensmallen optimizer OptimizerType . See https://www.ensmallen.org/docs.html#callback-documentation. |