[mlpack] Training very simple neural networks

[Marcus Edel]

Hello Kirill,

> After running this piece of code the predictedResponse matrix was the zero
> matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do
> wrong?

Depending on the network and input the ReLU function might be problematic since
a large gradient could cause the weights to update in such a way that the
network units will never activate. In your case, the Identity function might be
a better solution or the sigmoid function if you are going for logistic
regression. Using the following:

arma::mat data("1 2");
arma::mat trainingResponses("1 2; 3 4");

FFN<MeanSquaredError<>> ffn;
ffn.Add<Linear<>>(1, 2);
ffn.Add<IdentityLayer<> >();

ffn.Train(data, trainingResponses);

arma::mat predictedResponses;
ffn.Predict(data, predictedResponses);

I get the following results:

1.2766   1.8192
2.9841   4.0109

which is close to what you would expect.

> I also have noticed that if I don’t add ReLULayer<>, than there is an error
> during training:
> 
>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT
>   (application abort requested)

This is a shortcoming that occurs for a single layer network, in this case we
can't store the activation in the upcomming layer.

I hope this is helpful, let me know if I can clarify anything further.

Thanks,
Marcus

> On 29. May 2017, at 15:42, Kirill Mishchenko <ki.mishchenko at gmail.com> wrote:
> 
> Hi!
> 
> I’m working on cross-validation module for mlpack, and for better code coverage in tests I want to check some functionality on neural networks. For that I need to train a very simple feedforward neural network that is able to remember responses for training data. I tried the following:
> 
>   arma::mat data("1 2");
>   arma::mat trainingResponses("1 2; 3 4");
> 
>   FFN<MeanSquaredError<>> ffn;
>   ffn.Add<Linear<>>(1, 2);
>   ffn.Add<ReLULayer<>>();
> 
>   ffn.Train(data, trainingResponses);
> 
>   arma::mat predictedResponses;
>   ffn.Predict(data, predictedResponses);
> 
> After running this piece of code the predictedResponse matrix was the zero matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do wrong?
> 
> I also have noticed that if I don’t add ReLULayer<>, than there is an error during training:
> 
>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT (application abort requested)
> 
> Is it possible to train a linear model with the FNN class (i.e. linear regression)?
> 
> Best regards,
> 
> Kirill Mishchenko
> 
> 
> 
> _______________________________________________
> mlpack mailing list
> mlpack at lists.mlpack.org
> http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20170529/fc760839/attachment.html>