[mlpack] queries regarding the implementation of rnn.hpp in ann

Thu May 26 18:58:20 EDT 2016

Hello Nikhil,

I wish that we could have accepted everyone we wanted to, but at least I hope
that the application process was a good experience for you and hopefully kind of
enjoyable. I'm not aware of anyone who is actually working on the essential deep
learning module idea, so if you like to work on the project you a free to do so;
any contribution is more than welcome; mlpack is open-source, after all.

I like the Restricted Boltzmann Machine (RBM), so I would probably start with
that. Don't feel obligated to do so, there are a bunch of modules, you could
implement. Also, we can discuss, the details over the mailing list or use irc
#mlpack.

I hope this is helpful, let me know if I can clarify anything.

Thanks,
Marcus

> On 26 May 2016, at 20:36, Nikhil Yadala <nikhil.yadala at gmail.com> wrote:
> 
> Hi,
> 
>                            I have looked into,built, coded some algorithms with  the mlpack's API(especially the ann module), in March .I have also submitted a proposal (for GSoC) for the essential deep learning module, to implement API for deep neural nets (RBM, RNN,CNN etc).How ever, I have not got selected for the GSOC program, But I am still interested in contributing to MLpack. I would like to know, if any one has  already been working on implementing these modules(or had already implemented these modules) either as a part of GSOC project or individually . So that I could start working on this, If no one has started working on these. Would anyone here mind suggesting me which network I should start working on?
> 
> Thanks,
> Nikhil Yadala.
> 
> On Mon, Mar 14, 2016 at 7:18 PM, Marcus Edel <marcus.edel at fu-berlin.de <mailto:marcus.edel at fu-berlin.de>> wrote:
> Hello Nikhil,
> 
> > Could you tell me what the variables  ,inputSize, outputSize , seqOut specify
> 
> 
> I'll go and explain the parameter using a simple example. Imagine you have a
> sensor, that returns 3 values every second. You record the values for 10
> seconds. Every data point has the same class as label.
> 
> input = [4 3 9 5 8 9 1 2 5 9 0 0 0 0 8 ... ] labels = [1]
> 
> You use a recurrent neural network which takes 3 values as input and returns two
> values (class 1, class 2) as output. Since you are using a recurrent neural
> network you feed the network with a sequence (10 seconds x 3 values). The first
> step is to figure out the 'inputSize' of the network, which is the number of
> input units (3). We use that parameter to feed the network with the correct
> input at time 't' so (t=0 [4 3 9]) (t=1 [5 8 9]) ...
> 
> In this example, every input from the sequence belongs to the same class. But
> there are problems that require an output sequence. So you data could look like:
> 
> input = [4 3 9 5 8 9 1 2 5 9 0 0 0 0 8 ... ] labels = [1 2 1 2 2 ...]
> 
> So your input data at (t=0 input=[4 3 9]) (t=1 input=[5 8 9]) ... nothing has
> changed. But you expected output data changed (t=0 target=[1]) (t=1 target=[2])
> (t=2 target=[1]) ..
> 
> We use the 'outputSize' to use the right output at each time step.
> 
> The 'seqOut' is true if the network is trained to output a sequence (case 2) or
> to predict a single class (case 1).
> 
> > Also, As per what i understand regarding BPTT,each subsequence(time) (typically
> > k=3, at t-1,t,t+1) is considered one layer , and the rest all is similar to ffn
> > with a constraint that the weight matrix is same at every layer. But I dont
> > understand how the BPTT is implemented in rnn.hpp (If this is not the way , its
> > implemented here, could be please direct me to the link, where I could get a
> > better understanding of what BPTT does and how it does)
> 
> You are right, you feed the network with the input at time t and perform a
> forward pass. At the end of each forward pass you have to save the activations
> for the backward pass. You also have to set the input parameter of the recurrent
> layer, which is basically the activation of the hidden layer.
> 
> The backward pass is similar you first load the activations and then you can
> perform the normal backward pass with respect to the recurrent layer.
> 
> > Regarding the project proposal, I am planning to implement bidirectional deep
> > rnn, So, that there is no need to code the double layer brnn explicitly. , and
> > also to give very less time to implement convolutional auto-encoder, as the
> > cnn,hpp does almost the same thing , the only tweak that has to be done is to
> > hard code the outputlayer to inputs(am I wright? ) .Could you please give your
> > views on these?
> 
> You are right, you can use the rnn class and merge the two outputs at the end.
> It's also true, that the convolutional auto-encoder is straightforward to
> implement, but don't forget you have to write tests for each module.
> 
> I hope this is helpful, let me know if I can clarify anything.
> 
> Thanks,
> Marcus
> 
> 
> > On 14 Mar 2016, at 05:01, Nikhil Yadala <nikhil.yadala at gmail.com <mailto:nikhil.yadala at gmail.com>> wrote:
> >
> > Hi marcus,Ryan,
> >
> >                           I have gone through the complete code of ann, I don't get THE exact idea of how rnn is implemented in mlpack.I have a few queries
> >
> > Could you tell me what the variables  ,inputSize, outputSize , seqOut specify
> >
> > How is the output taken from the network, are we taking output after every time instance or is it that we take the output at the end of input (time) sequence.?
> >
> > Also, As per what i understand regarding BPTT,each subsequence(time) (typically k=3, at t-1,t,t+1) is considered one layer , and the rest all is similar to ffn with a constraint that the weight matrix is same at every layer. But I dont understand how the BPTT is implemented in rnn.hpp (If this is not the way , its implemented here, could be please direct me to the link, where I could get a better understanding of what BPTT does and how it does)
> >
> > Regarding the project proposal, I am planning to implement bidirectional deep rnn, So, that there is no need to code the double layer brnn explicitly. , and also to give very less time to implement convolutional auto-encoder, as the cnn,hpp does almost the same thing , the only tweak that has to be done is to hard code the outputlayer to inputs(am I wright? ) .Could you please give your views on these?
> >
> >
> > thanks,
> > Nikhil Yadala.
> > _______________________________________________
> > mlpack mailing list
> > mlpack at cc.gatech.edu <mailto:mlpack at cc.gatech.edu>
> > https://mailman.cc.gatech.edu/mailman/listinfo/mlpack <https://mailman.cc.gatech.edu/mailman/listinfo/mlpack>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160527/9257f7c3/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4972 bytes
Desc: not available
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160527/9257f7c3/attachment-0002.bin>