[mlpack] [mlpack/mlpack] Revamp ann modules (#831)

Marcus Edel notifications at github.com
Thu Dec 15 11:42:09 EST 2016 

This pull request revamps the ann modules. It should be a lot easier to create networks especially more complex networks that depend on the concat layer or the recurrent layer.

Changes:

- The way parameters of different types are dealt with is greatly simplified: arma::Mat<eT> for all input and output parameters instead of arma::Cube<eT> to represent 3rd order tensors. Makes it possible to e.g. use the linear layer for convolutional and standard feed forward networks.
- The introduced Recurrent module/layer is able to transform each non-recurrent layer/module into a recurrent layer.
- In addition to the feedforward, recurrent and convolution network test, each layer is now tested by approximating the Jacobian matrix.
- The introduction of the Concat module/layer allows us to split the network into subnetworks, which makes the implementation of e.g. the inception network a lot easier. (I'll open another PR for the inception layer once the PR is merged).

- The way the network is created is greatly simplified instead of using:

```
LinearLayer<> inputLayer(trainData.n_rows, hiddenLayerSize);
BiasLayer<> inputBiasLayer(hiddenLayerSize);
BaseLayer<PerformanceFunction> inputBaseLayer;
LinearLayer<> hiddenLayer1(hiddenLayerSize, trainLabels.n_rows);
BiasLayer<> hiddenBiasLayer1(trainLabels.n_rows);
BaseLayer<LogSoftMaxLayer<>> outputLayer;
OutputLayerType classOutputLayer;
auto modules = std::tie(inputLayer, inputBiasLayer, inputBaseLayer,
                        hiddenLayer1, hiddenBiasLayer1, outputLayer);

FFN<decltype(modules), decltype(classOutputLayer), RandomInitialization,
    PerformanceFunctionType> net(modules, classOutputLayer);
```
you can create the same network with:

```
FFN<NegativeLogLikelihood<> > model;
model.Add<Linear<> >(trainData.n_rows, hiddenLayerSize);
model.Add<SigmoidLayer<> >();
model.Add<Linear<> >(hiddenLayerSize, outputSize);
model.Add<LogSoftMax<> >();
```
You can view, comment on, or merge this pull request online at:

  https://github.com/mlpack/mlpack/pull/831

-- Commit Summary --

  * Fix explicitly specialized template issue.
  * Merge remote-tracking branch 'upstream/master'
  * Properly resetting auxBound. Start using a Reset() method, to avoid futures errors like this.
  * edge_boxes: feature extraction
  * backported ind2sub and sub2ind
  * backported ind2sub and sub2ind
  * Revert "edge_boxes: feature extraction"
  * backported sub2ind & ind2sub
  * Minor style fixes for ind2sub() test.
  * Add new contributors.
  * Try debugging symbols for AppVeyor build to see if it is faster.
  * Use appveyor cache (nuget and armadillo).
  * added test for ind2sub and sub2ind
  * fix doc tutorial
  * fix typo
  * Merge remote-tracking branch 'upstream/master'
  * Merge remote-tracking branch 'upstream/master'
  * Merge remote-tracking branch 'upstream/master'
  * Remove the RMVA model.
  * Remove unused ann functions.
  * Remove unused ann layer.
  * Increase the number of template arguments for the boost list class.
  * Move pooling rules into the pooling class. So that we can use the MaxPooling and MeanPooling class names for the actual module name.
  * Use the stride parameter inside the convolution function.
  * Increase the number of template arguments for the boost list class.
  * Remove stride paramater from svd and fft convolution rule.
  * Refactor ann layer.
  * Remove the rmva model for the CmakeLists file.
  * Add visitor function set; which abstracts away the different types of layers.
  * Minor style fixes.
  * Refactor recurrent network test.
  * Remove unused pooling test.
  * Refactor FNN class; works for CNNs and FFNs
  * Refactor RNN class; works will all current modules including the updated recurrent module.
  * Include all layer modules.
  * Minor style fixes.
  * Add layer traits to check for the input width, height and model function.
  * Refactor neural visual attention modules.
  * Use refactored rnn,ffn classes for the ann tests.
  * Add ann module test.
  * Split layer modules into definition and implementation.
  * Merge remote-tracking branch 'upstream/master'
  * Remove the RMVA model.
  * Remove unused ann functions.
  * Remove unused ann layer.
  * Increase the number of template arguments for the boost list class.
  * Move pooling rules into the pooling class. So that we can use the MaxPooling and MeanPooling class names for the actual module name.
  * Use the stride parameter inside the convolution function.
  * Increase the number of template arguments for the boost list class.
  * Remove stride paramater from svd and fft convolution rule.
  * Refactor ann layer.
  * Remove the rmva model for the CmakeLists file.
  * Add visitor function set; which abstracts away the different types of layers.
  * Minor style fixes.
  * Refactor recurrent network test.
  * Remove unused pooling test.
  * Refactor FNN class; works for CNNs and FFNs
  * Refactor RNN class; works will all current modules including the updated recurrent module.
  * Include all layer modules.
  * Minor style fixes.
  * Add layer traits to check for the input width, height and model function.
  * Refactor neural visual attention modules.
  * Use refactored rnn,ffn classes for the ann tests.
  * Add ann module test.
  * Split layer modules into definition and implementation.
  * Merge branch 'ann' of github.com:zoq/mlpack into ann

-- File Changes --

    M src/mlpack/methods/CMakeLists.txt (1)
    M src/mlpack/methods/ann/CMakeLists.txt (6)
    D src/mlpack/methods/ann/cnn.hpp (448)
    D src/mlpack/methods/ann/cnn_impl.hpp (289)
    M src/mlpack/methods/ann/convolution_rules/fft_convolution.hpp (10)
    M src/mlpack/methods/ann/convolution_rules/naive_convolution.hpp (58)
    M src/mlpack/methods/ann/convolution_rules/svd_convolution.hpp (8)
    M src/mlpack/methods/ann/ffn.hpp (398)
    M src/mlpack/methods/ann/ffn_impl.hpp (461)
    M src/mlpack/methods/ann/layer/CMakeLists.txt (77)
    R src/mlpack/methods/ann/layer/add.hpp (96)
    A src/mlpack/methods/ann/layer/add_impl.hpp (68)
    A src/mlpack/methods/ann/layer/add_merge.hpp (139)
    A src/mlpack/methods/ann/layer/add_merge_impl.hpp (61)
    M src/mlpack/methods/ann/layer/base_layer.hpp (63)
    D src/mlpack/methods/ann/layer/bias_layer.hpp (208)
    D src/mlpack/methods/ann/layer/binary_classification_layer.hpp (106)
    A src/mlpack/methods/ann/layer/concat.hpp (186)
    A src/mlpack/methods/ann/layer/concat_impl.hpp (159)
    A src/mlpack/methods/ann/layer/concat_performance.hpp (120)
    A src/mlpack/methods/ann/layer/concat_performance_impl.hpp (118)
    R src/mlpack/methods/ann/layer/constant.hpp (57)
    A src/mlpack/methods/ann/layer/constant_impl.hpp (65)
    D src/mlpack/methods/ann/layer/conv_layer.hpp (324)
    A src/mlpack/methods/ann/layer/convolution.hpp (344)
    A src/mlpack/methods/ann/layer/convolution_impl.hpp (328)
    A src/mlpack/methods/ann/layer/dropconnect.hpp (209)
    A src/mlpack/methods/ann/layer/dropconnect_impl.hpp (118)
    D src/mlpack/methods/ann/layer/dropconnect_layer.hpp (361)
    R src/mlpack/methods/ann/layer/dropout.hpp (126)
    A src/mlpack/methods/ann/layer/dropout_impl.hpp (84)
    R src/mlpack/methods/ann/layer/glimpse.hpp (318)
    A src/mlpack/methods/ann/layer/glimpse_impl.hpp (224)
    A src/mlpack/methods/ann/layer/hard_tanh.hpp (140)
    A src/mlpack/methods/ann/layer/hard_tanh_impl.hpp (72)
    D src/mlpack/methods/ann/layer/hard_tanh_layer.hpp (259)
    R src/mlpack/methods/ann/layer/join.hpp (63)
    A src/mlpack/methods/ann/layer/join_impl.hpp (60)
    A src/mlpack/methods/ann/layer/layer.hpp (30)
    M src/mlpack/methods/ann/layer/layer_traits.hpp (33)
    A src/mlpack/methods/ann/layer/layer_types.hpp (117)
    A src/mlpack/methods/ann/layer/layer_visitor.hpp (1034)
    A src/mlpack/methods/ann/layer/layer_visitor_impl.hpp (1241)
    R src/mlpack/methods/ann/layer/leaky_relu.hpp (99)
    A src/mlpack/methods/ann/layer/leaky_relu_impl.hpp (60)
    R src/mlpack/methods/ann/layer/linear.hpp (129)
    A src/mlpack/methods/ann/layer/linear_impl.hpp (87)
    D src/mlpack/methods/ann/layer/linear_layer.hpp (289)
    A src/mlpack/methods/ann/layer/linear_no_bias.hpp (154)
    A src/mlpack/methods/ann/layer/linear_no_bias_impl.hpp (83)
    R src/mlpack/methods/ann/layer/log_softmax.hpp (73)
    A src/mlpack/methods/ann/layer/log_softmax_impl.hpp (85)
    R src/mlpack/methods/ann/layer/lookup.hpp (110)
    A src/mlpack/methods/ann/layer/lookup_impl.hpp (74)
    A src/mlpack/methods/ann/layer/lstm.hpp (237)
    A src/mlpack/methods/ann/layer/lstm_impl.hpp (273)
    D src/mlpack/methods/ann/layer/lstm_layer.hpp (418)
    A src/mlpack/methods/ann/layer/max_pooling.hpp (280)
    A src/mlpack/methods/ann/layer/max_pooling_impl.hpp (149)
    A src/mlpack/methods/ann/layer/mean_pooling.hpp (252)
    A src/mlpack/methods/ann/layer/mean_pooling_impl.hpp (126)
    A src/mlpack/methods/ann/layer/mean_squared_error.hpp (100)
    A src/mlpack/methods/ann/layer/mean_squared_error_impl.hpp (57)
    D src/mlpack/methods/ann/layer/multiclass_classification_layer.hpp (98)
    R src/mlpack/methods/ann/layer/multiply_constant.hpp (44)
    A src/mlpack/methods/ann/layer/multiply_constant_impl.hpp (51)
    R src/mlpack/methods/ann/layer/negative_log_likelihood.hpp (77)
    A src/mlpack/methods/ann/layer/negative_log_likelihood_impl.hpp (76)
    D src/mlpack/methods/ann/layer/one_hot_layer.hpp (96)
    D src/mlpack/methods/ann/layer/pooling_layer.hpp (267)
    A src/mlpack/methods/ann/layer/recurrent.hpp (213)
    A src/mlpack/methods/ann/layer/recurrent_attention.hpp (263)
    A src/mlpack/methods/ann/layer/recurrent_attention_impl.hpp (204)
    A src/mlpack/methods/ann/layer/recurrent_impl.hpp (206)
    R src/mlpack/methods/ann/layer/reinforce_normal.hpp (64)
    A src/mlpack/methods/ann/layer/reinforce_normal_impl.hpp (69)
    A src/mlpack/methods/ann/layer/select.hpp (111)
    A src/mlpack/methods/ann/layer/select_impl.hpp (75)
    A src/mlpack/methods/ann/layer/sequential.hpp (203)
    A src/mlpack/methods/ann/layer/sequential_impl.hpp (154)
    D src/mlpack/methods/ann/layer/sparse_bias_layer.hpp (177)
    D src/mlpack/methods/ann/layer/sparse_output_layer.hpp (227)
    R src/mlpack/methods/ann/layer/vr_class_reward.hpp (130)
    A src/mlpack/methods/ann/layer/vr_class_reward_impl.hpp (101)
    D src/mlpack/methods/ann/network_traits.hpp (55)
    D src/mlpack/methods/ann/network_util.hpp (247)
    D src/mlpack/methods/ann/network_util_impl.hpp (286)
    D src/mlpack/methods/ann/performance_functions/CMakeLists.txt (17)
    D src/mlpack/methods/ann/performance_functions/cee_function.hpp (74)
    D src/mlpack/methods/ann/performance_functions/mse_function.hpp (61)
    D src/mlpack/methods/ann/performance_functions/sparse_function.hpp (141)
    D src/mlpack/methods/ann/performance_functions/sse_function.hpp (64)
    D src/mlpack/methods/ann/pooling_rules/CMakeLists.txt (15)
    D src/mlpack/methods/ann/pooling_rules/max_pooling.hpp (56)
    D src/mlpack/methods/ann/pooling_rules/mean_pooling.hpp (56)
    M src/mlpack/methods/ann/rnn.hpp (749)
    M src/mlpack/methods/ann/rnn_impl.hpp (525)
    D src/mlpack/methods/rmva/CMakeLists.txt (17)
    D src/mlpack/methods/rmva/rmva.hpp (963)
    D src/mlpack/methods/rmva/rmva_impl.hpp (0)
    D src/mlpack/methods/rmva/rmva_main.cpp (0)
    I src/mlpack/prereqs.hpp (0)
    I src/mlpack/tests/CMakeLists.txt (0)
    I src/mlpack/tests/activation_functions_test.cpp (0)
    A src/mlpack/tests/ann_layer_test.cpp (0)
    I src/mlpack/tests/convolution_test.cpp (0)
    I src/mlpack/tests/convolutional_network_test.cpp (0)
    I src/mlpack/tests/feedforward_network_test.cpp (0)
    D src/mlpack/tests/network_util_test.cpp (0)
    D src/mlpack/tests/performance_functions_test.cpp (0)
    D src/mlpack/tests/pooling_rules_test.cpp (0)
    I src/mlpack/tests/recurrent_network_test.cpp (0)

-- Patch Links --

https://github.com/mlpack/mlpack/pull/831.patch
https://github.com/mlpack/mlpack/pull/831.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/mlpack/mlpack/pull/831
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20161215/23b79ea0/attachment-0001.html>

More information about the mlpack mailing list