[mlpack] Training very simple neural networks

[Kirill Mishchenko]

Thank you, Marcus, it works.

Yet another question: is there any optimizer that let me make responses of FFN arbitrarily close to what I expect (arma::mat("1 2; 3 4”))?

Best regards,

Kirill Mishchenko


> On 29 May 2017, at 19:30, Marcus Edel <marcus.edel at fu-berlin.de> wrote:
> 
> Hello Kirill,
> 
>> After running this piece of code the predictedResponse matrix was the zero
>> matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do
>> wrong?
> 
> Depending on the network and input the ReLU function might be problematic since
> a large gradient could cause the weights to update in such a way that the
> network units will never activate. In your case, the Identity function might be
> a better solution or the sigmoid function if you are going for logistic
> regression. Using the following:
> 
> arma::mat data("1 2");
> arma::mat trainingResponses("1 2; 3 4");
> 
> FFN<MeanSquaredError<>> ffn;
> ffn.Add<Linear<>>(1, 2);
> ffn.Add<IdentityLayer<> >();
> 
> ffn.Train(data, trainingResponses);
> 
> arma::mat predictedResponses;
> ffn.Predict(data, predictedResponses);
> 
> I get the following results:
> 
> 1.2766   1.8192
> 2.9841   4.0109
> 
> which is close to what you would expect.
> 
>> I also have noticed that if I don’t add ReLULayer<>, than there is an error
>> during training:
>> 
>>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT
>>   (application abort requested)
> 
> This is a shortcoming that occurs for a single layer network, in this case we
> can't store the activation in the upcomming layer.
> 
> I hope this is helpful, let me know if I can clarify anything further.
> 
> Thanks,
> Marcus
> 
>> On 29. May 2017, at 15:42, Kirill Mishchenko <ki.mishchenko at gmail.com <mailto:ki.mishchenko at gmail.com>> wrote:
>> 
>> Hi!
>> 
>> I’m working on cross-validation module for mlpack, and for better code coverage in tests I want to check some functionality on neural networks. For that I need to train a very simple feedforward neural network that is able to remember responses for training data. I tried the following:
>> 
>>   arma::mat data("1 2");
>>   arma::mat trainingResponses("1 2; 3 4");
>> 
>>   FFN<MeanSquaredError<>> ffn;
>>   ffn.Add<Linear<>>(1, 2);
>>   ffn.Add<ReLULayer<>>();
>> 
>>   ffn.Train(data, trainingResponses);
>> 
>>   arma::mat predictedResponses;
>>   ffn.Predict(data, predictedResponses);
>> 
>> After running this piece of code the predictedResponse matrix was the zero matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do wrong?
>> 
>> I also have noticed that if I don’t add ReLULayer<>, than there is an error during training:
>> 
>>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT (application abort requested)
>> 
>> Is it possible to train a linear model with the FNN class (i.e. linear regression)?
>> 
>> Best regards,
>> 
>> Kirill Mishchenko
>> 
>> 
>> 
>> _______________________________________________
>> mlpack mailing list
>> mlpack at lists.mlpack.org <mailto:mlpack at lists.mlpack.org>
>> http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20170529/1c5a2763/attachment-0001.html>