[mlpack] An Introduction

Thu Mar 24 01:49:39 EDT 2016

Dear Marcus,

unfortunately, I realize that my other commitments during the summer will
not allow me to participate in Googles Summer of Code. Yet, I still wish to
contribute to the development of mlpack, especially in the projects that I
had mentioned (Neuro Evolution, Deep Learning, Le Net). How would you
suggest I coordinate with those who will be, or are already, working on the
projects? Can I still hope for guidance from the mentors? I hope to make
long term contributions, and become a part of the mlpack family. I believe
that I can help in not insignificant ways.

Sincerely,
Amitrajit.

On Sun, Mar 13, 2016 at 6:22 PM, Marcus Edel <marcus.edel at fu-berlin.de>
wrote:

> Hello Amitrajit,
>
> 1. Referring to the organization of mlpack, should the neuroevolution
> algorithms
> be implemented as an 'optimizer' within 'ann' 'methods' (supporting
> 'optimize',
> 'reset' and 'update')? Or should they be implemented as separate 'methods'?
>
>
> The interface you are referring to is the old ann optimizer interface we
> recently switch to the mlpack core optimizer interface. See
> https://github.com/mlpack/mlpack/issues/555 for more information. I don't
> think
> that you could use the evolution optimizer to train other models. That
> said, I
> think we should implement the neuroevolution algorithms as separate methods
> using a structure that looks like the cf/amf framework.
>
> 2. Keeping track of the historical markers for NEAT suggests a linked list,
> connectionist ANN representation. Should this be implemented as a separate
> datastructure, or serve as the connectionist representation of the network
> itself? If a vectorized matrix representation is used, how will new nodes
> evolved between layers be maintained?
>
>
> I think you are right, a connectionist representation is the superior
> solution
> here. I guess we could also a matrix representation, and increase the
> matrix
> size when evolving the network. I would prefer to serve the connectionist
> representation of the network itself. The goal should be to reuse the
> network
> model for the various evolution methods e.g. NEAT, HyperNEAT, etc. Take a
> look
> at the
> http://www.mlpack.org/docs/mlpack-2.0.1/doxygen.php?doc=amftutorial.html
> tutorial to get an impression about how we could reuse parts of the amf/cf
> structure to model the evolution project.
>
> 3. What functions will the deep learning modules support? 'Train',
> 'predict',
> 'evaluate', 'gradient' and 'serialize'? Some deep learning networks use
> forms of
> unsupervised pretraining. How should these by incorporated into the mlpack
> API?
>
>
> Exactly, once you use the mlpack core optimizer you model has to expose an
> 'Evaluate' and 'Gradient' function. Also, regarding the design guidelines
> each
> machine learning algorithms that have a specific training step should
> provide a
> 'Train' and 'Predict' function. For more information take a look at:
> https://github.com/mlpack/mlpack/wiki/DesignGuidelines
>
> One way to incorporate pretraining unsupervised or supervised is to e.g.
> train a
> Deep Belief Network (there isn't a DBN implementation right now, I'll just
> use
> that as an example) extract the weights, using the 'Parameter()''
> function, to
> initialize the other model. Another way could be to provide a constructor
> that
> takes a model, an pretrains the parameter. You could use the
> 'NetworkWeights(parameter, network)' function to propagate the pretrained
> parameter through the network.
>
>
> I hope this is helpful, let me know if I can clarify anything.
>
> Thanks,
> Marcus
>
> On 11 Mar 2016, at 19:56, Amitrajit Sarkar <aaiijmrtt at gmail.com> wrote:
>
> Dear Marcus,
>
> I apologize for the delay. I have been through all the material that you
> suggested: thank you. I have also been through the structure of mlpack. I
> have a few questions:
>
> 1. Referring to the organization of mlpack, should the neuroevolution
> algorithms be implemented as an 'optimizer' within 'ann' 'methods'
> (supporting 'optimize', 'reset' and 'update')? Or should they be
> implemented as separate 'methods'?
>
> 2. Keeping track of the historical markers for NEAT suggests a linked
> list, connectionist ANN representation. Should this be implemented as a
> separate datastructure, or serve as the connectionist representation of the
> network itself? If a vectorized matrix representation is used, how will new
> nodes evolved between layers be maintained?
>
> 3. What functions will the deep learning modules support? 'Train',
> 'predict', 'evaluate', 'gradient' and 'serialize'? Some deep learning
> networks use forms of unsupervised pretraining. How should these by
> incorporated into the mlpack API?
>
> Best,
> Amitrajit.
>
> On Tue, Mar 8, 2016 at 3:18 AM, Marcus Edel <marcus.edel at fu-berlin.de>
> wrote:
>
>> Hello Amitrajit,
>>
>> I shall certainly look through the new references I find on the reading
>> lists
>> before filling out my application. If there are any resources in
>> particular that
>> you would like me to take note of, do mention them.
>>
>>
>> Here are some papers for the Neuroevolution idea. A good theoretical
>> understanding of what these models do and why they work is a necessity to
>> be
>> able to implement these well.
>>
>> - HyperNEAT-GGP:
>> http://nn.cs.utexas.edu/downloads/papers/hausknecht.gecco12.pdf
>> - NEAT:
>> http://nn.cs.utexas.edu/?stanley:ec02
>> - CMA-ES:
>> http://image.diku.dk/igel/paper/NfRLUES.pdf
>> - CoSyNE:
>> ftp://ftp.cs.utexas.edu/pub/neural-nets/papers/gomez.ecml06.ps
>> - Multi-Objective Neuroevolution in Super Mario Bros.:
>> http://www.diva-portal.org/smash/get/diva2:676807/FULLTEXT01.pdf
>>
>> And here are some papers for the 'We need to go deeper' idea.
>>
>> - Going Deeper with Convolutions:
>> http://arxiv.org/abs/1409.4842
>> - Selective Search for Object Recognition:
>> http://koen.me/research/pub/uijlings-ijcv2013-draft.pdf
>> - Scalable Object Detection using Deep Neural Networks (multi-box):
>> http://arxiv.org/abs/1312.2249
>>
>> And here are some papers on neural network models.
>>
>> Restricted Boltzmann Machines (RBM)
>> - https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf
>> - http://deeplearning.net/tutorial/rbm.html
>>
>> Deep Belief Networks (DBN)
>> - http://www.cs.toronto.edu/~rsalakhu/papers/science.pdf
>> - http://deeplearning.net/tutorial/DBN.html
>>
>> Radial Basis Function Networks (RBFN)
>> - http://www.cc.gatech.edu/~isbell/tutorials/rbf-intro.pdf
>>
>> Bidrectional Recurrent networks (BRN)
>> Note: mlpack provides already an implementation for recurrent network
>> - http://www.di.ufpe.br/~fnj/RNA/bibliografia/BRNN.pdf
>>
>> Convolutional Auto-Encoders (CAE)
>> - http://people.idsia.ch/~masci/papers/2011_icann.pdf
>>
>> Hopfield neural networks (HNN)
>> - http://page.mi.fu-berlin.de/rojas/neural/chapter/K13.pdf
>>
>> Keep in mind that you don't have to implement all of this models; A good
>> project
>> will select a handful of architectures and implement them with tests and
>> documentation. Writing good tests is often the hardest part, so keep that
>> in
>> mind when you create your project timeline.
>>
>> I was wondering whether a connectionist approach would be better with
>> regard to
>> implementing the Neuroevolution algorithms when dealing with Augmenting
>> Topologies. I would like your views on the matter.
>>
>>
>> Basically it's a for performance reasons, but you can mimic a
>> connectionist
>> model, by simply setting the weights in the LinearLayer to zero , so that
>> unit_11^(0) is only connected with unit_11^(1) and unit_12^(1)
>> and not with unit_13^(1). You can also implement a special Layer to get
>> this
>> done even more easily.
>>
>> Also, would you like to see a basic implementation of CNE, using the
>> existing
>> mlpack neural networks, as a warm-up task? I really look forward to
>> contributing
>> to mlpack.
>>
>>
>> Contributing is not a requirement for an application. Anyway, If you like
>> to do
>> that as warm-up task, I'm here to help you out. Keep in mind that you
>> have to
>> write a test, before I can merge anything in.
>>
>> Thanks,
>> Marcus
>>
>>
>> On 07 Mar 2016, at 19:40, Amitrajit Sarkar <aaiijmrtt at gmail.com> wrote:
>>
>> Hello Marcus,
>>
>> I agree: each of these projects requires a lot of background study.
>> However, my undergrad research work has been focused on neural networks and
>> deep learning for over a year now. Hence I am already familiar with the
>> concepts appearing on the Ideas page, as well as those previously mentioned
>> in the mailing list, having implemented several myself. I shall certainly
>> look through the new references I find on the reading lists before filling
>> out my application. If there are any resources in particular that you would
>> like me to take note of, do mention them.
>>
>> I built mlpack from source, tried the tutorials, and started deciphering
>> the source code. I understand that neural networks in mlpack use armadillo
>> matrices for efficiency, a vectorized approach. I was wondering whether a
>> connectionist approach would be better with regard to implementing the
>> Neuroevolution algorithms when dealing with Augmenting Topologies. I would
>> like your views on the matter.
>>
>> Also, would you like to see a basic implementation of CNE, using the
>> existing mlpack neural networks, as a warm-up task? I really look forward
>> to contributing to mlpack.
>>
>> Regards,
>> Amitrajit.
>>
>> On Mon, Mar 7, 2016 at 5:38 AM, Marcus Edel <marcus.edel at fu-berlin.de>
>> wrote:
>>
>>> Hello Amitrajit,
>>>
>>> sorry for the slow response.
>>>
>>> I am especially interested in:
>>>
>>> Neuroevolution Algorithms,
>>> Essential Deep Learning Modules,
>>> We Need To Go Deeper - Google LeNet.
>>>
>>>
>>> I might suggest that you narrow your focus because each of these
>>> projects has a
>>> significant amount of background knowledge that is necessary.
>>>
>>> To learn more about each of the projects than what has been listed on
>>> the Ideas
>>> page, take a look at the mailing list archives:
>>>
>>> https://mailman.cc.gatech.edu/pipermail/mlpack/
>>>
>>> However, others are already working on the warmup tasks listed alongside
>>> the
>>> projects. Are there any other tasks that I could try?
>>>
>>>
>>> Don't worry, contributing is not a requirement for an application. So if
>>> you
>>> don't find anything that you think you can do, that's not necessarily a
>>> problem.
>>> However, I'll see if I can add some more "easy" issues in the next
>>> couple of
>>> days. On the other side, you are always welcome to just poke around the
>>> library
>>> and try to fix any problems you find, or improve the speed of various
>>> parts.
>>>
>>> Thanks,
>>> Marcus
>>>
>>> On 06 Mar 2016, at 08:39, Amitrajit Sarkar <aaiijmrtt at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I am Amitrajit Sarkar, a CS undergrad from Jadavpur University, India. I
>>> have been working on machine learning for over a year now. I even have my
>>> own neural networks library <https://github.com/aaiijmrtt/NET>, which I
>>> wrote from scratch while trying to understand existing theories. I am very
>>> eager to contribute to mlpack for GSoC 2016, as almost all the projects
>>> excite me equally.
>>>
>>> I am especially interested in:
>>>
>>> Neuroevolution Algorithms,
>>> <https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#neuroevolution-algorithms>
>>> Essential Deep Learning Modules,
>>> <https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#essential-deep-learning-modules>
>>> We Need To Go Deeper - Google LeNet.
>>> <https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#we-need-to-go-deeper---googlenet>
>>>
>>> I have implemented basic neuroevolution algorithms here
>>> <https://github.com/aaiijmrtt/LEARNING>, and several deep learning
>>> modules here <https://github.com/aaiijmrtt/NET>. I am certain that I
>>> can take up the tasks. However, others are already working on the warmup
>>> tasks listed alongside the projects. Are there any other tasks that I could
>>> try? I have a lot of experience with research work, and am a skilled coder.
>>>
>>> I am attaching my CV for reference. You may find more about my interests
>>> on my blog <http://aaiijmrtt.github.io/>.
>>>
>>> Cheers,
>>> Amitrajit.
>>> <cv.pdf>_______________________________________________
>>> mlpack mailing list
>>> mlpack at cc.gatech.edu
>>> https://mailman.cc.gatech.edu/mailman/listinfo/mlpack
>>>
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160324/0126b1d2/attachment-0002.html>