[mlpack] Parallel Stochastic Potimazation Methods

[Ryan Curtin]

On Wed, Mar 16, 2016 at 08:50:06PM +0200, pvachill at ece.auth.gr wrote:
> Hello!
> 
> I have implemented a naive parallel edition of the serial sgd implementation
> that already exists in mlpack build, using pthreads. My goal is to make
> HOGWILD! algorithm. I send my code and  an example using the test_function.
> Before i continue implementing HOGWILD! algorithm i would like you to
> mention if i am in the correct way.I would also appreciate if someone could
> send me a sparse dataset as mentioned at this paper:
> http://papers.nips.cc/paper/4390-hogwild-a-lock-free-approach-to-parallelizing-stochastic-gradient-descent.pdf.

Hi Axilleas,

Thanks for the code.  Could you please use OpenMP instead of pthreads?
I think that this problem is solvable with OpenMP, and OpenMP gives much
cleaner and easier to read code.

> From what i have already understand, i could make some suggestions which are
> still a bit of fuzzy in my mind:
> 1)Create stepSize an abstract class that would have different
> implementations for different approaches

This would be a good idea; you could add a "StepStrategy" template
parameter, and then the user could choose something like a constant step
size, a decreasing step size, and so forth.

> 2)Change Optimize member function to template function for different
> approaches
> 3)Create functions that don't use datasets as an abstract class and a second
> abstract class for those who use datasets.
> 4)Also, change sgd to abstract sgd by the same reasoning.

Can you explain more of what you mean by these suggestions?  I'm not
sure I understand them.

You can find large sparse datasets at the MovieLens website:
http://grouplens.org/datasets/movielens/

But there is a little more work that you'll have to do to test Hogwild!
in the manner suggested in the paper; you'll have to write a function
class that implements the objective function for matrix decomposition,
and then use that function with your parallel SGD class to test.

Let me know if I can clarify anything.

Thanks!

Ryan

-- 
Ryan Curtin    | "Gentlemen, you can't fight in here!  This is the
ryan at ratml.org | War Room!" - President Muffley