[mlpack] [GSOC '16] : Student interested in Low rank/sparse optimization using Frank-Wolfe

[Ryan Curtin]

On Fri, Mar 11, 2016 at 05:20:30PM +0000, Kwon, Jin Kyoung wrote:
> Dear Ryan Curtin,
> 
> (This may be a duplicate - I sent the same email from my gmail
> account, which I suspect got filtered by the mailing list.. so I am
> resending it from my email registered w/ the mailing list) 
> 
> Thank you so much for your quick and thorough reply! I had some
> follow-up questions, and would really appreciate if you could spend a
> moment to help me understand the library better.
> 
> I saw that there are documentations for optimizers at
> http://mlpack.org/doxygen/, albeit they are for V1.0.12. Would
> documentation of optimizer architecture for V2.0.1 be of interest to
> you as something I can do as a part of the project? Anyhow, looking
> through doxygen still gave me a good idea of the architecture. I
> understood that wrapping optimizers in *_function.hpp (e. g.
> aug_lagrangian_function wrapping aug_lagrangian) enables it to be used
> by other optimizers. What I noticed were the absenses of test
> functions for some optimizers (e.g. sa, sgd). Is there a reason for
> this? Also, is there a reason for naming tests under aug_lagrangian to
> be “aug_lagrangian_test_functions” versus just “test_functions” (as
> done under lbfgs and sgd)?

Ah, you should probably look at this documentation instead:

http://mlpack.org/docs/mlpack-git/doxygen.php

The other documentation was very old and I've removed it from the
website.

If you would like to document the OptimizerType and FunctionType policy
classes in a document kind of like
http://mlpack.org/docs/mlpack-git/doxygen.php?doc=metrics.html
or http://mlpack.org/docs/mlpack-git/doxygen.php?doc=trees.html I'd be
happy to take a look over a PR and work with you to get it into the main
codebase.

I think the OptimizerType API is pretty straightforward and so is the
FunctionType API, so it shouldn't be too much work.

The test functions for the Augmented Lagrangian method are constrained,
so, they are a little different than the other optimizer test functions.
If you want to rename that file to test_functions.hpp, feel free.

> I would also like to look at how the artificial neural network code
> were modified to use the optimizers. This would be help me better
> understand how to carry out your suggestion for modifying programs
> like mlpack_softmax_regression to fit the Frank-Wolfe optimizer. From
> which version to another did the nn code change? Also, where may I
> find the nn code in the library? I suspect it is under
> src/mlpack/methods/sparse_coding.. but I also saw that we used to have
> ann module, which I do not see in the current version.

The neural network code can be found in src/mlpack/methods/ann/.  Looks
like the changes happened here:

https://github.com/mlpack/mlpack/commit/69f97e50aeddfb83e708475606b74a05bee8034a#diff-c3a396ee194aa467a2e2441643b1aaf7
https://github.com/mlpack/mlpack/commit/f6c27ed38633f732d1e8dd34e29e4be79b9d9f87#diff-c3a396ee194aa467a2e2441643b1aaf7
https://github.com/mlpack/mlpack/commit/43f0f13fe84e11709b9d6254ad11827c6eaed80b#diff-c3a396ee194aa467a2e2441643b1aaf7

> I do agree - Charles is a great lecturer! He hasn't mentioned mlpack -
> what brought me here google queries for "georgia tech machine
> learning", etc :) I do think he should include mlpack as a part of
> "Suggest Software" under our class syllabus! We could suggest the
> idea, as I suspect there must be C++ users in the class. As of now we
> have an assignment on implementing optimizers such as randomized hill
> climbing (RHC), MIMIC, SA and genetic algorithms (GA). Would
> implementing RHC, MIMIC and GA be of future interests for mlpack?

I can't seem to find the syllabus online, but I think it would be a good
idea if you would suggest to him that mlpack should be put on the
list... :)

If you have the flexibility in your assignment to implement those
optimizers to satisfy the mlpack OptimizerType interface, and if you
write some tests for them, then I'd be happy to accept them into mlpack.
It shouldn't be too hard to write tests for them.  Writing those
optimizers for mlpack would probably require much more general code than
what you'd write for the assignment, though.

Thanks,

Ryan

-- 
Ryan Curtin    | "Why is it that the landscape is moving... but the boat is
ryan at ratml.org | still?"  - Train Driver