[mlpack] Guidance for the GSOC for Deep learning modules and tree traversers

[Nikhil Yadala]

Hi marcus,

                          I have gone through the complete code of ann, I
don't get THE exact idea of how rnn is implemented in mlpack.I have a few
queries

Could you tell me what the variables  ,inputSize, outputSize , seqOut
specify

How is the output taken from the network, are we taking output after every
time instance or is it that we take the output at the end of input (time)
sequence.?

Also, As per what i understand regarding BPTT,each subsequence(time)
(typically k=3, at t-1,t,t+1) is considered one layer , and the rest all is
similar to ffn with a constraint that the weight matrix is same at every
layer. But I dont understand how the BPTT is implemented in rnn.hpp (If
this is not the way , its implemented here, could be please direct me to
the link, where I could get a better understanding of what BPTT does and
how it does)

Regarding the project proposal, I am planning to implement bidirectional
deep rnn, So, that there is no need to code the double layer brnn
explicitly. , and also to give very less time to implement convolutional
auto-encoder, as the cnn,hpp does almost the same thing , the only tweak
that has to be done is to hard code the outputlayer to inputs(am I wright?
) .Could you please give your views on these?


thanks,
Nikhil Yadala.



On Tue, Mar 8, 2016 at 2:43 AM, Marcus Edel <marcus.edel at fu-berlin.de>
wrote:

> Hello nikhil,
>
> Don't worry, contributing is not a requirement for an application. So if
> you
> don't find anything that you think you can do, that's not necessarily a
> problem.
> But I guess, it's a good opportunity to get familiar with the codebase.
> You can
> always ask me, to clarify the purpose of a particular function. However,
> I'll
> see if I can add some more "easy" issues in the next couple of days, but
> often
> things like that get taken care of quickly so it's hard to keep a good
> backlog
> for people who want to contribute. On the other side, you are always
> welcome to
> just poke around the library and try to fix any problems you find, or
> improve
> the speed of various parts.
>
> Let me know if you have any more questions that I can answer.
>
> Thanks,
> Marcus
>
> On 07 Mar 2016, at 21:10, Nikhil Yadala <nikhil.yadala at gmail.com> wrote:
>
> Hi marcus,
> Sorry for the late reply, I was having major exams till yesterday.
> I have gone through the ffn code and now i understand the flow of the code
> , but i have a problem in understanding what each function does(though i
> have a clarity , whats happening over all). Could you give some beginner
> tasks to be solved regarding this part of code,(if possible),?
>
> thanks,
> Nikhil Yadala,
> Btech CSE.
>
> On Mon, Feb 22, 2016 at 9:26 PM, Marcus Edel <marcus.edel at fu-berlin.de>
> wrote:
>
>> Hello nikhil,
>>
>> that sounds good. It would probably be useful to read many papers on
>> different
>> basic neural network models you like to implement over the summer,
>> including the
>> design guidelines and the current neural network code (take a look at the
>> FFN
>> class) that details the abstractions on which mlpack network models are
>> based:
>>
>> - https://github.com/mlpack/mlpack/wiki/DesignGuidelines
>> -
>> https://github.com/mlpack/mlpack/blob/master/src/mlpack/methods/ann/ffn.hpp
>> -
>> https://github.com/mlpack/mlpack/blob/master/src/mlpack/methods/ann/ffn_impl.hpp
>>
>> And here are some papers on neural network models. A good theoretical
>> understanding of what these models do and why they work is a necessity to
>> be
>> able to implement these well.
>>
>> Restricted Boltzmann Machines (RBM)
>> - https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf
>> - http://deeplearning.net/tutorial/rbm.html
>>
>> Deep Belief Networks (DBN)
>> - http://www.cs.toronto.edu/~rsalakhu/papers/science.pdf
>> - http://deeplearning.net/tutorial/DBN.html
>>
>> Radial Basis Function Networks (RBFN)
>> - http://www.cc.gatech.edu/~isbell/tutorials/rbf-intro.pdf
>>
>> Bidrectional Recurrent networks (BRN)
>> Note: mlpack provides already an implementation for recurrent network
>> - http://www.di.ufpe.br/~fnj/RNA/bibliografia/BRNN.pdf
>>
>> Convolutional Auto-Encoders (CAE)
>> - http://people.idsia.ch/~masci/papers/2011_icann.pdf
>>
>> Hopfield neural networks (HNN)
>> - http://page.mi.fu-berlin.de/rojas/neural/chapter/K13.pdf
>>
>> Keep in mind that you don't have to implement all of this models; A good
>> project
>> will select a handful of architectures and implement them with tests and
>> documentation. Writing good tests is often the hardest part, so keep that
>> in
>> mind when you create your project timeline.
>>
>> Let me know if you have any more questions that I can answer.
>>
>> Thanks
>> Marcus
>>
>> On 21 Feb 2016, at 12:50, Nikhil Yadala <nikhil.yadala at gmail.com> wrote:
>>
>> hi marcus,
>>
>>       Thanks for the reply. As you have said i have gone through the
>> information in the above links. I would like to stick for the
>> implementation of deep learning modules project.
>>
>>     I have known the basics of deep learning and i have read some
>> chapters of the book you have mentioned.Should I be knowing all the
>> concepts of deep learning?, if yes, i guess i should complete reading the
>> book, Nevertheless I am now good at understanding the things given in the
>> language of deep learning.Could you tell me what is exactly required, so
>> that i would concentrate on that topic to a deeper extent.?
>>
>> thanks,
>> nikhil yadala.
>>
>> On Wed, Feb 17, 2016 at 8:01 PM, Marcus Edel <marcus.edel at fu-berlin.de>
>> wrote:
>>
>>> Hello Nikhil,
>>>
>>> What I really like about the "Essential Deep Learning Modules" project
>>> is that
>>> it offers the chance to learn about various fundamental deep learning
>>> models
>>> from a practical perspective. As the usual suspects are the most
>>> commonly used
>>> algorithms by the community, it is likely that people and developer will
>>> execute
>>> your code or use the code that you wrote as basis for their own
>>> structures.
>>>
>>> Since you already build mlpack, one place to start is to look at the
>>> tutorials
>>> and try to compile and run simple mlpack programs.
>>>
>>> http://www.mlpack.org/tutorials.html
>>>
>>> Then, you could look at the list of issues on Github and maybe you can
>>> find an
>>> easy and interesting bug to solve.
>>>
>>> https://github.com/mlpack/mlpack/issues
>>>
>>> The issues are labeled with difficulty, so you may be able to find some
>>> which are suited to your level of experience with mlpack.
>>>
>>> The literature for the Essential Deep Learning Modules project depends
>>> on the
>>> network models you like to implement over the summer. But here are some
>>> hopefully useful links to get information about the basics:
>>>
>>> - "Deep learning reading list" (http://deeplearning.net/reading-list/)
>>> - "Neural Networks for Machine Learning" by Geoffrey Hinton (
>>> https://www.coursera.org/course/neuralnets)
>>> - "Deep learning" by Yoshua Bengio, Ian Goodfellow and Aaron Courville (
>>> http://www.deeplearningbook.org/)
>>>
>>> If you have found network models you're interested in, you
>>> could start reading about them and we can go from there.
>>>
>>> Since Ryan is the mentor of the "Improvement of tree traversers" and the
>>> expert,
>>> I go and just make a quote here:
>>>
>>> "A good place to start is by working through the mlpack tutorials and
>>> making sure you can get mlpack to compile and understand how to use it.
>>> Once you've done that, you should probably read about the ideas behind
>>> dual-tree algorithms, since your interest is in the 'improvement of tree
>>> traversers' project.  You might start here:
>>>
>>>
>>> http://machinelearning.wustl.edu/mlpapers/paper_files/icml2013_curtin13.pdf
>>>
>>> There are a lot of references in that document, and you should probably
>>> read most of them to get a good idea of what is going on (especially the
>>> cover tree paper by Langford)."
>>>
>>> Since you said you are working on a python to c++ translator, you might
>>> be also
>>> interested in the automatic bindings project. That project would give a
>>> way to
>>> automatically generate MATLAB, Python, and R bindings, which would be
>>> much
>>> easier to maintain than having to maintain each binding individually:
>>>
>>>
>>> https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#automatic-bindings
>>>
>>> Let me know if you have any more questions that I can answer.
>>>
>>> Thanks!
>>> Marcus
>>>
>>> On 16 Feb 2016, at 22:40, Nikhil Yadala <nikhil.yadala at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>>            I am nikhil yadala, pursuing 2nd year of my Btech at dept of
>>> CSE at Indian Institute of Technology Guwahati( IIT Guwahati). I am very
>>> much interested in machine learning particularly in computational biology.
>>> Currently i am doing research in Gene profiling using deep learning methods.
>>>
>>>           I am interested in two of the projects that are floated as
>>> potential ones for the GSOC,2016
>>>           1)Improvement of Tree traversers
>>>           2)Essential Deep Learning Modules
>>> I have enough expertise in C,C++,python,Currently iam developing a
>>> PYTHON TO C++ TRNSLATOR In c++.
>>> I would be glad if any one over there would guide me as to get enough
>>> knowledge to get going with these projects. I have already built mlpack
>>> over my ubuntu and started to understand the code.
>>>
>>> Thanks,
>>> NIkhil yadala
>>>
>>> _______________________________________________
>>> mlpack mailing list
>>> mlpack at cc.gatech.edu
>>> https://mailman.cc.gatech.edu/mailman/listinfo/mlpack
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160313/6ee04290/attachment-0002.html>