[mlpack] Training very simple neural networks

[Kirill Mishchenko]

Ok, thank you for your advices. Setting the number of iterations to 0 is probably what I need.

Best regards,

Kirill Mishchenko

> On 30 May 2017, at 15:00, Marcus Edel <marcus.edel at fu-berlin.de> wrote:
> 
> Hello Kirill,
> 
> If you use the MeanSquaredError as performance function you can get
> arbitrarily close to the expected mean squared error. But it's not granted you
> reach the expected error with the specified architecture or it might take some
> time to get there, so you have to experiment with some models that might
> converge in an acceptable time. I would expect that:
> 
> FFN<MeanSquaredError<>> ffn;
> ffn.Add<IdentityLayer<> >();
> ffn.Add<Linear<>>(1, 8);
> ffn.Add<SigmoidLayer<> >();
> ffn.Add<Linear<>>(8, 2);
> ffn.Add<IdentityLayer<> >();
> 
> returns better results in terms of the mean squared error. Anyway, most of the
> optimizer classes define a maxIterations parameter that will limit the number of
> iterations. If set to 0 the optimizer does not return before the specified
> tolerance is reached.
> 
> I hope this is helpful, if you have any further questions don't hesitate to ask.
> 
> Thanks,
> Marcus
> 
>> On 29. May 2017, at 17:42, Kirill Mishchenko <ki.mishchenko at gmail.com <mailto:ki.mishchenko at gmail.com>> wrote:
>> 
>> Thank you, Marcus, it works.
>> 
>> Yet another question: is there any optimizer that let me make responses of FFN arbitrarily close to what I expect (arma::mat("1 2; 3 4”))?
>> 
>> Best regards,
>> 
>> Kirill Mishchenko
>> 
>> 
>>> On 29 May 2017, at 19:30, Marcus Edel <marcus.edel at fu-berlin.de <mailto:marcus.edel at fu-berlin.de>> wrote:
>>> 
>>> Hello Kirill,
>>> 
>>>> After running this piece of code the predictedResponse matrix was the zero
>>>> matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do
>>>> wrong?
>>> 
>>> Depending on the network and input the ReLU function might be problematic since
>>> a large gradient could cause the weights to update in such a way that the
>>> network units will never activate. In your case, the Identity function might be
>>> a better solution or the sigmoid function if you are going for logistic
>>> regression. Using the following:
>>> 
>>> arma::mat data("1 2");
>>> arma::mat trainingResponses("1 2; 3 4");
>>> 
>>> FFN<MeanSquaredError<>> ffn;
>>> ffn.Add<Linear<>>(1, 2);
>>> ffn.Add<IdentityLayer<> >();
>>> 
>>> ffn.Train(data, trainingResponses);
>>> 
>>> arma::mat predictedResponses;
>>> ffn.Predict(data, predictedResponses);
>>> 
>>> I get the following results:
>>> 
>>> 1.2766   1.8192
>>> 2.9841   4.0109
>>> 
>>> which is close to what you would expect.
>>> 
>>>> I also have noticed that if I don’t add ReLULayer<>, than there is an error
>>>> during training:
>>>> 
>>>>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT
>>>>   (application abort requested)
>>> 
>>> This is a shortcoming that occurs for a single layer network, in this case we
>>> can't store the activation in the upcomming layer.
>>> 
>>> I hope this is helpful, let me know if I can clarify anything further.
>>> 
>>> Thanks,
>>> Marcus
>>> 
>>>> On 29. May 2017, at 15:42, Kirill Mishchenko <ki.mishchenko at gmail.com <mailto:ki.mishchenko at gmail.com>> wrote:
>>>> 
>>>> Hi!
>>>> 
>>>> I’m working on cross-validation module for mlpack, and for better code coverage in tests I want to check some functionality on neural networks. For that I need to train a very simple feedforward neural network that is able to remember responses for training data. I tried the following:
>>>> 
>>>>   arma::mat data("1 2");
>>>>   arma::mat trainingResponses("1 2; 3 4");
>>>> 
>>>>   FFN<MeanSquaredError<>> ffn;
>>>>   ffn.Add<Linear<>>(1, 2);
>>>>   ffn.Add<ReLULayer<>>();
>>>> 
>>>>   ffn.Train(data, trainingResponses);
>>>> 
>>>>   arma::mat predictedResponses;
>>>>   ffn.Predict(data, predictedResponses);
>>>> 
>>>> After running this piece of code the predictedResponse matrix was the zero matrix (2x2) rather than something close to arma::mat("1 2; 3 4”). What did I do wrong?
>>>> 
>>>> I also have noticed that if I don’t add ReLULayer<>, than there is an error during training:
>>>> 
>>>>   unknown location:0: fatal error: in "CVTest/MSENNTest": signal: SIGABRT (application abort requested)
>>>> 
>>>> Is it possible to train a linear model with the FNN class (i.e. linear regression)?
>>>> 
>>>> Best regards,
>>>> 
>>>> Kirill Mishchenko
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> mlpack mailing list
>>>> mlpack at lists.mlpack.org <mailto:mlpack at lists.mlpack.org>
>>>> http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack <http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack>
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20170530/014091d7/attachment-0001.html>