[mlpack] queries regarding the implementation of rnn.hpp in ann

[Marcus Edel]

Hello Nikhil,

> Could you tell me what the variables  ,inputSize, outputSize , seqOut specify


I'll go and explain the parameter using a simple example. Imagine you have a
sensor, that returns 3 values every second. You record the values for 10
seconds. Every data point has the same class as label.

input = [4 3 9 5 8 9 1 2 5 9 0 0 0 0 8 ... ] labels = [1]

You use a recurrent neural network which takes 3 values as input and returns two
values (class 1, class 2) as output. Since you are using a recurrent neural
network you feed the network with a sequence (10 seconds x 3 values). The first
step is to figure out the 'inputSize' of the network, which is the number of
input units (3). We use that parameter to feed the network with the correct
input at time 't' so (t=0 [4 3 9]) (t=1 [5 8 9]) ...

In this example, every input from the sequence belongs to the same class. But
there are problems that require an output sequence. So you data could look like:

input = [4 3 9 5 8 9 1 2 5 9 0 0 0 0 8 ... ] labels = [1 2 1 2 2 ...]

So your input data at (t=0 input=[4 3 9]) (t=1 input=[5 8 9]) ... nothing has
changed. But you expected output data changed (t=0 target=[1]) (t=1 target=[2])
(t=2 target=[1]) ..

We use the 'outputSize' to use the right output at each time step.

The 'seqOut' is true if the network is trained to output a sequence (case 2) or
to predict a single class (case 1).

> Also, As per what i understand regarding BPTT,each subsequence(time) (typically
> k=3, at t-1,t,t+1) is considered one layer , and the rest all is similar to ffn
> with a constraint that the weight matrix is same at every layer. But I dont
> understand how the BPTT is implemented in rnn.hpp (If this is not the way , its
> implemented here, could be please direct me to the link, where I could get a
> better understanding of what BPTT does and how it does)

You are right, you feed the network with the input at time t and perform a
forward pass. At the end of each forward pass you have to save the activations
for the backward pass. You also have to set the input parameter of the recurrent
layer, which is basically the activation of the hidden layer.

The backward pass is similar you first load the activations and then you can
perform the normal backward pass with respect to the recurrent layer.

> Regarding the project proposal, I am planning to implement bidirectional deep
> rnn, So, that there is no need to code the double layer brnn explicitly. , and
> also to give very less time to implement convolutional auto-encoder, as the
> cnn,hpp does almost the same thing , the only tweak that has to be done is to
> hard code the outputlayer to inputs(am I wright? ) .Could you please give your
> views on these?

You are right, you can use the rnn class and merge the two outputs at the end.
It's also true, that the convolutional auto-encoder is straightforward to
implement, but don't forget you have to write tests for each module.

I hope this is helpful, let me know if I can clarify anything.

Thanks,
Marcus


> On 14 Mar 2016, at 05:01, Nikhil Yadala <nikhil.yadala at gmail.com> wrote:
> 
> Hi marcus,Ryan,
> 
>                           I have gone through the complete code of ann, I don't get THE exact idea of how rnn is implemented in mlpack.I have a few queries 
> 
> Could you tell me what the variables  ,inputSize, outputSize , seqOut specify 
> 
> How is the output taken from the network, are we taking output after every time instance or is it that we take the output at the end of input (time) sequence.?
> 
> Also, As per what i understand regarding BPTT,each subsequence(time) (typically k=3, at t-1,t,t+1) is considered one layer , and the rest all is similar to ffn with a constraint that the weight matrix is same at every layer. But I dont understand how the BPTT is implemented in rnn.hpp (If this is not the way , its implemented here, could be please direct me to the link, where I could get a better understanding of what BPTT does and how it does)
> 
> Regarding the project proposal, I am planning to implement bidirectional deep rnn, So, that there is no need to code the double layer brnn explicitly. , and also to give very less time to implement convolutional auto-encoder, as the cnn,hpp does almost the same thing , the only tweak that has to be done is to hard code the outputlayer to inputs(am I wright? ) .Could you please give your views on these?
> 
> 
> thanks,
> Nikhil Yadala.
> _______________________________________________
> mlpack mailing list
> mlpack at cc.gatech.edu
> https://mailman.cc.gatech.edu/mailman/listinfo/mlpack

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5233 bytes
Desc: not available
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160314/1f3c5942/attachment-0002.bin>