[mlpack] Training very simple neural networks

[Marcus Edel]

Hello Kirill,

If you use the MeanSquaredError as performance function you can get
arbitrarily close to the expected mean squared error. But it's not granted you
reach the expected error with the specified architecture or it might take some
time to get there, so you have to experiment with some models that might
converge in an acceptable time. I would expect that:

FFN<MeanSquaredError<>> ffn;
ffn.Add<IdentityLayer<> >();
ffn.Add<Linear<>>(1, 8);
ffn.Add<SigmoidLayer<> >();
ffn.Add<Linear<>>(8, 2);
ffn.Add<IdentityLayer<> >();

returns better results in terms of the mean squared error. Anyway, most of the
optimizer classes define a maxIterations parameter that will limit the number of
iterations. If set to 0 the optimizer does not return before the specified
tolerance is reached.

I hope this is helpful, if you have any further questions don't hesitate to ask.

Thanks,
Marcus

> On 29. May 2017, at 17:42, Kirill Mishchenko <ki.mishchenko at gmail.com> wrote:
> 
> Thank you, Marcus, it works.
> 
> Yet another question: is there any optimizer that let me make responses of FFN arbitrarily close to what I expect (arma::mat("1 2; 3 4”))?
> 
> Best regards,
> 
> Kirill Mishchenko
> 
> 
>> On 29 May 2017, at 19:30, Marcus Edel <marcus.edel at fu-berlin.de <mailto:marcus.edel at fu-berlin.de>> wrote:
>> 
>> Hello Kirill,
>> 
>>> After running this piece of code the predictedResponse matrix was the zero
>>> matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do
>>> wrong?
>> 
>> Depending on the network and input the ReLU function might be problematic since
>> a large gradient could cause the weights to update in such a way that the
>> network units will never activate. In your case, the Identity function might be
>> a better solution or the sigmoid function if you are going for logistic
>> regression. Using the following:
>> 
>> arma::mat data("1 2");
>> arma::mat trainingResponses("1 2; 3 4");
>> 
>> FFN<MeanSquaredError<>> ffn;
>> ffn.Add<Linear<>>(1, 2);
>> ffn.Add<IdentityLayer<> >();
>> 
>> ffn.Train(data, trainingResponses);
>> 
>> arma::mat predictedResponses;
>> ffn.Predict(data, predictedResponses);
>> 
>> I get the following results:
>> 
>> 1.2766   1.8192
>> 2.9841   4.0109
>> 
>> which is close to what you would expect.
>> 
>>> I also have noticed that if I don’t add ReLULayer<>, than there is an error
>>> during training:
>>> 
>>>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT
>>>   (application abort requested)
>> 
>> This is a shortcoming that occurs for a single layer network, in this case we
>> can't store the activation in the upcomming layer.
>> 
>> I hope this is helpful, let me know if I can clarify anything further.
>> 
>> Thanks,
>> Marcus
>> 
>>> On 29. May 2017, at 15:42, Kirill Mishchenko <ki.mishchenko at gmail.com <mailto:ki.mishchenko at gmail.com>> wrote:
>>> 
>>> Hi!
>>> 
>>> I’m working on cross-validation module for mlpack, and for better code coverage in tests I want to check some functionality on neural networks. For that I need to train a very simple feedforward neural network that is able to remember responses for training data. I tried the following:
>>> 
>>>   arma::mat data("1 2");
>>>   arma::mat trainingResponses("1 2; 3 4");
>>> 
>>>   FFN<MeanSquaredError<>> ffn;
>>>   ffn.Add<Linear<>>(1, 2);
>>>   ffn.Add<ReLULayer<>>();
>>> 
>>>   ffn.Train(data, trainingResponses);
>>> 
>>>   arma::mat predictedResponses;
>>>   ffn.Predict(data, predictedResponses);
>>> 
>>> After running this piece of code the predictedResponse matrix was the zero matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do wrong?
>>> 
>>> I also have noticed that if I don’t add ReLULayer<>, than there is an error during training:
>>> 
>>>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT (application abort requested)
>>> 
>>> Is it possible to train a linear model with the FNN class (i.e. linear regression)?
>>> 
>>> Best regards,
>>> 
>>> Kirill Mishchenko
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> mlpack mailing list
>>> mlpack at lists.mlpack.org <mailto:mlpack at lists.mlpack.org>
>>> http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack <http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack>
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20170530/7f0b0f95/attachment.html>