[mlpack] An Introduction

Mon Mar 7 16:48:58 EST 2016

Hello Amitrajit,

> I shall certainly look through the new references I find on the reading lists
> before filling out my application. If there are any resources in particular that
> you would like me to take note of, do mention them.

Here are some papers for the Neuroevolution idea. A good theoretical
understanding of what these models do and why they work is a necessity to be
able to implement these well.

- HyperNEAT-GGP:
	http://nn.cs.utexas.edu/downloads/papers/hausknecht.gecco12.pdf
- NEAT:
	http://nn.cs.utexas.edu/?stanley:ec02
- CMA-ES:
	http://image.diku.dk/igel/paper/NfRLUES.pdf
- CoSyNE:
	ftp://ftp.cs.utexas.edu/pub/neural-nets/papers/gomez.ecml06.ps
- Multi-Objective Neuroevolution in Super Mario Bros.: 
	http://www.diva-portal.org/smash/get/diva2:676807/FULLTEXT01.pdf

And here are some papers for the 'We need to go deeper' idea.

- Going Deeper with Convolutions:
	http://arxiv.org/abs/1409.4842
- Selective Search for Object Recognition:
	http://koen.me/research/pub/uijlings-ijcv2013-draft.pdf
- Scalable Object Detection using Deep Neural Networks (multi-box):
	http://arxiv.org/abs/1312.2249

And here are some papers on neural network models. 

Restricted Boltzmann Machines (RBM)
- https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf
- http://deeplearning.net/tutorial/rbm.html

Deep Belief Networks (DBN)
- http://www.cs.toronto.edu/~rsalakhu/papers/science.pdf
- http://deeplearning.net/tutorial/DBN.html

Radial Basis Function Networks (RBFN)
- http://www.cc.gatech.edu/~isbell/tutorials/rbf-intro.pdf

Bidrectional Recurrent networks (BRN)
Note: mlpack provides already an implementation for recurrent network
- http://www.di.ufpe.br/~fnj/RNA/bibliografia/BRNN.pdf

Convolutional Auto-Encoders (CAE)
- http://people.idsia.ch/~masci/papers/2011_icann.pdf

Hopfield neural networks (HNN)
- http://page.mi.fu-berlin.de/rojas/neural/chapter/K13.pdf

Keep in mind that you don't have to implement all of this models; A good project
will select a handful of architectures and implement them with tests and
documentation. Writing good tests is often the hardest part, so keep that in
mind when you create your project timeline.

> I was wondering whether a connectionist approach would be better with regard to
> implementing the Neuroevolution algorithms when dealing with Augmenting
> Topologies. I would like your views on the matter.

Basically it's a for performance reasons, but you can mimic a connectionist
model, by simply setting the weights in the LinearLayer to zero , so that
unit_11^(0) is only connected with unit_11^(1) and unit_12^(1)
and not with unit_13^(1). You can also implement a special Layer to get this
done even more easily.

> Also, would you like to see a basic implementation of CNE, using the existing
> mlpack neural networks, as a warm-up task? I really look forward to contributing
> to mlpack.

Contributing is not a requirement for an application. Anyway, If you like to do
that as warm-up task, I'm here to help you out. Keep in mind that you have to
write a test, before I can merge anything in.

Thanks,
Marcus

> On 07 Mar 2016, at 19:40, Amitrajit Sarkar <aaiijmrtt at gmail.com> wrote:
> 
> Hello Marcus,
> 
> I agree: each of these projects requires a lot of background study. However, my undergrad research work has been focused on neural networks and deep learning for over a year now. Hence I am already familiar with the concepts appearing on the Ideas page, as well as those previously mentioned in the mailing list, having implemented several myself. I shall certainly look through the new references I find on the reading lists before filling out my application. If there are any resources in particular that you would like me to take note of, do mention them.
> 
> I built mlpack from source, tried the tutorials, and started deciphering the source code. I understand that neural networks in mlpack use armadillo matrices for efficiency, a vectorized approach. I was wondering whether a connectionist approach would be better with regard to implementing the Neuroevolution algorithms when dealing with Augmenting Topologies. I would like your views on the matter.
> 
> Also, would you like to see a basic implementation of CNE, using the existing mlpack neural networks, as a warm-up task? I really look forward to contributing to mlpack.
> 
> Regards,
> Amitrajit.
> 
> On Mon, Mar 7, 2016 at 5:38 AM, Marcus Edel <marcus.edel at fu-berlin.de <mailto:marcus.edel at fu-berlin.de>> wrote:
> Hello Amitrajit,
> 
> sorry for the slow response.
> 
>> I am especially interested in:
>> 
>> Neuroevolution Algorithms,
>> Essential Deep Learning Modules,
>> We Need To Go Deeper - Google LeNet.
> 
> 
> I might suggest that you narrow your focus because each of these projects has a
> significant amount of background knowledge that is necessary.
> 
> To learn more about each of the projects than what has been listed on the Ideas
> page, take a look at the mailing list archives:
> 
> https://mailman.cc.gatech.edu/pipermail/mlpack/ <https://mailman.cc.gatech.edu/pipermail/mlpack/>
> 
>> However, others are already working on the warmup tasks listed alongside the
>> projects. Are there any other tasks that I could try?
> 
> Don't worry, contributing is not a requirement for an application. So if you
> don't find anything that you think you can do, that's not necessarily a problem.
> However, I'll see if I can add some more "easy" issues in the next couple of
> days. On the other side, you are always welcome to just poke around the library
> and try to fix any problems you find, or improve the speed of various parts.
> 
> Thanks,
> Marcus
> 
>> On 06 Mar 2016, at 08:39, Amitrajit Sarkar <aaiijmrtt at gmail.com <mailto:aaiijmrtt at gmail.com>> wrote:
>> 
>> Hi,
>> 
>> I am Amitrajit Sarkar, a CS undergrad from Jadavpur University, India. I have been working on machine learning for over a year now. I even have my own neural networks library <https://github.com/aaiijmrtt/NET>, which I wrote from scratch while trying to understand existing theories. I am very eager to contribute to mlpack for GSoC 2016, as almost all the projects excite me equally.
>> 
>> I am especially interested in:
>> 
>> Neuroevolution Algorithms, <https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#neuroevolution-algorithms>
>> Essential Deep Learning Modules, <https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#essential-deep-learning-modules>
>> We Need To Go Deeper - Google LeNet. <https://github.com/mlpack/mlpack/wiki/SummerOfCodeIdeas#we-need-to-go-deeper---googlenet>
>> 
>> I have implemented basic neuroevolution algorithms here <https://github.com/aaiijmrtt/LEARNING>, and several deep learning modules here <https://github.com/aaiijmrtt/NET>. I am certain that I can take up the tasks. However, others are already working on the warmup tasks listed alongside the projects. Are there any other tasks that I could try? I have a lot of experience with research work, and am a skilled coder.
>> 
>> I am attaching my CV for reference. You may find more about my interests on my blog <http://aaiijmrtt.github.io/>.
>> 
>> Cheers,
>> Amitrajit.
>> <cv.pdf>_______________________________________________
>> mlpack mailing list
>> mlpack at cc.gatech.edu <mailto:mlpack at cc.gatech.edu>
>> https://mailman.cc.gatech.edu/mailman/listinfo/mlpack <https://mailman.cc.gatech.edu/mailman/listinfo/mlpack>
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160307/8309c1e6/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5233 bytes
Desc: not available
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20160307/8309c1e6/attachment-0002.bin>