[mlpack] GSoC 2017 wrap-up

[Ryan Curtin]

Hello there,

Today the results of GSoC 2017 were announced, and I am pleased to
report that all ten of our students this summer passed!  Thank you to
all the students and mentors for your hard work---I think that there
were some really cool contributions that will be a part of the next
major mlpack release, and I am hopeful that all students had a great
experience and continue to contribute to (or help maintain) mlpack in
the future.

Thanks also to the mentors---Marcus, Stephen, and Mikhail.  It's a lot
of work to mentor!

Here's a short list of the projects and what the students accomplished:

----
Chenzhe Diao: "Frank-Wolfe Algorithm for Sparse Optimization and
               Regularization of Atom Norms"
http://mlpack.org/gsocblog/frankwolfe-algorithm-and-solving-sparse-solutions-of-atom-norm-constraint-problems.html

This summer, Chenzhe implemented the Frank-Wolfe algorithm, found in
src/mlpack/core/optimizers/fw/.  The class is templatized, so its
behavior is pretty controllable.  In addition to other problems, the FW
optimizer can be used for orthogonal matching pursuit, which Chenzhe
also implemented.  Soon Chenzhe will open a PR to incorporate the FW
optimizer into the matrix completion code (in
src/mlpack/methods/matrix_completion/).

----
Dewang Sultania: "Better Benchmarking for mlpack Libraries"
http://mlpack.org/gsocblog/benchmarking-mlpack-libraries.html

Dewang worked all summer on updating the benchmarking scripts for each
library and also adding new libraries, such as milk, R, annoy, mrpt, and
dlib-ml, as well as adding new scripts for different machine learning
methods.  This is a nice improvement to the benchmarking system, which
is now more usable as a result.

----
Kartik Nighania: "Neural Evolution Algorithms on NES Games"
http://mlpack.org/gsocblog/summary-of-evolution-algorithms.html

Kartik spent the summer implementing different neural evolution
algorithms that can be used to learn to play NES games.  The CMAES and
CNE optimizers were implemented as general mlpack optimizers.  The CNE
optimizer was merged to src/mlpack/core/optimizers/cne/, and the CMAES
optimizer is finishing review now and will probably be merged soon.
Kartik is continuing to work on the NEAT optimizer.

----
Kirill Mishchenko: "Cross-Validation and Hyper-Parameter Tuning"
http://mlpack.org/gsocblog/cross-validation-and-hyper-parameter-tuning-summary.html

Kirill created a cross-validation and hyper-parameter tuning
infrastructure that makes heavy use of template metaprogramming.  It has
been merged and can be found in src/mlpack/core/cv/ and
src/mlpack/core/hpt/.  A tutorial will be merged soon, and then the
hyper-parameter tuner should be ready to use.  The tuner also supports
using different optimizers than just grid search or random search, and
should be very useful for anyone using mlpack to find the optimal
parameters for their algorithm.

----
Konstantin Sidorov: "Augmented Recurrent Neural Networks"
http://mlpack.org/gsocblog/augmented-recurrent-neural-networks-summary.html

Konstantin's work centered around complex memory models for neural
networks.  He implemented a benchmarking framework for memory-based
tasks (such as sorting, copy, and addition tasks) which was merged and
also some utility programs in the models/ repository.  Since neural
networks are tricky and difficult, there was a lot of debugging and
tuning to get the results we expected.  Currently the implementation of
the hierarchical memory unit (HAM) is nearly complete and I hope to see
this merged reasonably soon.

----
Kris Singh: "Deep Learning Modules"
http://mlpack.org/gsocblog/deep-learning-modules-summary.html

Kris and Mikhail worked tirelessly all summer to implement a number of
deep learning techniques---RBMs, ssRBMs, GANs, and other useful things
like a ResizeLayer and a CrossEntropyWithLogits layer.  I think a good
amount of this code will be merged very soon.  The details in Kris' post
(above) are really interesting, especially about how much tuning is
needed to get these techniques to work as well as the papers.  It's very
cool to see such a highly discussed technique like GANs getting this
level of attention in mlpack.

----
Saurabh Gupta: "Build Testing with Docker and VMs"
http://masterblaster.mlpack.org/job/docker%20mlpack%20nightly%20build/

This summer Saurabh worked on setting up a better build and testing
environment for mlpack.  Using the nice build server 'masterblaster'
that we have, he automatically built Docker containers for a huge number
of combinations of compiler version, Armadillo version, and Boost
version.  This makes testing a lot easier, and the link above is to the
new 'mlpack nightly build' that runs entirely in Docker.  This really
helps us get better test coverage on different setups.  Some work is
continuing on getting OS X virtual machines running in Docker, which
would also augment our testing capabilities.

----
Shangtong Zhang: "Deep Reinforcement Learning Methods"
http://mlpack.org/gsocblog/deep-reinforcement-learning-methods-summary.html

Shangtong's work this summer focused on reinforcement learning.  He
implemented popular reinforcement learning algorithms like double
q-learning, n-step q learning, asynchronous one-step SARSA, and others.
In addition, Shangtong also spent a lot of time discussing and updating
core mlpack neural network abstractions, most recently with the Alias
layer idea, which allows a user to share weights across multiple
networks.  Most of the work can be found in
methods/reinforcement_learning/.

----
Shikhar Bhardwaj: "Profiling for Parallelization and Parallel Stochastic
                   Optimization Methods"
http://mlpack.org/gsocblog/profiling-for-parallelization-and-parallel-stochastic-optimization-methods-summary.html

This summer Shikhar worked with Yannis to make several improvements to
mlpack.  First, he overhauled the testing framework to use CTest (part
of CMake).  Then, he worked to make both NaiveKMeans and SGD parallel
via OpenMP, which can give a pretty nice speed boost.  Lastly, he
implemented the stochastic coordinate descent (SCD) and Hogwild!
optimizers.  SCD can outperform other optimizers for certain tasks, and
that implementation should be merged shortly.

----
Sumedh Ghaisas: "Neural Turing Machines"
http://mlpack.org/gsocblog/neural-turing-machines-final-summary.html

Sumedh, who is our first two-time GSoC student participant, worked all
summer to implement Neural Turing Machines, which are a fascinating way
to add external memory to a neural network.  In addition, he implemented
GRUs (gated recurrent units), another technique like LSTMs for memory in
a neural network, and is currently working on batch normalization
layers.  The Neural Turing Machine is complete and should be merged
soon.  I think that once it is merged, a lot of really exciting networks
can be built.

----

So, I would say it was a very productive summer!  I don't have the exact
numbers, but mlpack contained ~100k lines of code at the beginning of
the summer, and now contains ~115k (that's a very rough estimate...).
That doesn't include all the PRs that are still yet to be merged.  But
in any case, whatever the real number is, I am still amazed to see what
is accomplished each summer during GSoC.  Congratulations to all
students (and mentors)!  You worked hard and the value of the work is
clear.  I cannot wait to release all of this cool new functionality.

(Also, I noticed that 80% of our students this year had names starting
with K or S.  I am not sure how that happened!)

-- 
Ryan Curtin    | "It's too bad she won't live!  But then again, who
ryan at ratml.org | does?" - Gaff