[mlpack] GSoC 2014 simulated annealing optimizer

[Ryan Curtin]

On Mon, Jul 07, 2014 at 12:37:42PM -0500, Zhihao Lou wrote:
> Hi Ryan,
> 
> I've look through the code in the truck and all your changes look fine to
> me.
> 
> On the other side, I've catched 2 small bugs in my own submission:
> oldEnergy in SA::Optimize should be double rather than size_t. On the
> stopping condition, the annealer should let the move control have a chance
> to correct move size before stopping. I've attached my patch.

Great, thanks!  I've applied the patch in r16786.

> Also, my website is http://people.cs.uchicago.edu/~zhlou/ if you want to
> include that on the list of contributors.

Ok; I have added your name to core.hpp and also on the website:
http://www.mlpack.org/about.html

> By the way, when I trying to compile the code checked out from trunk, the
> compilation failed and spitted out the following message:
> /users/zhlou/mlpack-sa/src/mlpack/tests/svd_test.cpp: In member function
> ‘void SVDBatchTest::SVDMomentumTest::test_method()’:
> /users/zhlou/mlpack-sa/src/mlpack/tests/svd_test.cpp:45: error: reference
> to ‘math’ is ambiguous
> /users/zhlou/usr/include/armadillo_bits/constants_compat.hpp:158: error:
> candidates are: typedef class arma::Math<double> arma::math
> /users/zhlou/mlpack-sa/src/mlpack/../mlpack/core/math/clamp.hpp:14: error:
>                 namespace mlpack::math { }
> 
> The compilation is happened on a CentOS 6 with gcc 4.4.7 and my custom
> compiled boost and armadillo so this may not be replicated on other
> configuration. My workaround is to specify mlpack::math for all 4 cases
> inside svd_test.cpp.

I applied your workaround to svd_test.cpp in r16785.  Thanks for
pointing it out!

> On Thu, Jul 3, 2014 at 1:10 PM, Zhihao Lou <lzh1984 at gmail.com> wrote:
> > Ahh... You caught me right before I'm set for the independence day break.
> > I'll take a look at the code next week.
> >
> > And yes it's my honor to be listed as one of the contributors.
> >
> > Now onto the search space. I can think of two ways of working with bounded
> > search space.
> >
> > One is to let the function being optimized simply return something like
> > DBL_MAX (or numeric_limits<double>::max() if you prefer c++ style) when any
> > parameter go outside of the search space. This way there is no change
> > needed for the annealing code. I've used this method in one of my problems
> > and I can report that at least in my problem, I haven't seen any instances
> > that repeatedly hit the boundary. In addition, it will allow the user to
> > switch between hard and soft bound (like a penalty term) with relatively
> > less changes.

I think letting the function define its bounds is probably the best way
to approach the problem.  Otherwise, we would have to modify the
existing optimizers to deal with the search boundaries.  So it is
probably simpler to allow the user to define a bounded function by
something like

double Evaluate(const arma::mat& coordinates)
{
  if (OutsideBoundaries(coordinates))
    return DBL_MAX;
  else
    return ActuallyEvaluate(coordinates);
}

Do you think that's reasonable?  If you have a better solution, let me
know, but I think that's the best way to go for now.

I hope your break went well. :)

Thanks,

Ryan

-- 
Ryan Curtin    | "Because for the people who knew what it was, it
ryan at ratml.org | would be attacking their brain."