[mlpack] GSoC 2014 simulated annealing optimizer

[Zhihao Lou]

Hi Ryan,

I've look through the code in the truck and all your changes look fine to
me.

On the other side, I've catched 2 small bugs in my own submission:
oldEnergy in SA::Optimize should be double rather than size_t. On the
stopping condition, the annealer should let the move control have a chance
to correct move size before stopping. I've attached my patch.

Also, my website is http://people.cs.uchicago.edu/~zhlou/ if you want to
include that on the list of contributors.

By the way, when I trying to compile the code checked out from trunk, the
compilation failed and spitted out the following message:
/users/zhlou/mlpack-sa/src/mlpack/tests/svd_test.cpp: In member function
‘void SVDBatchTest::SVDMomentumTest::test_method()’:
/users/zhlou/mlpack-sa/src/mlpack/tests/svd_test.cpp:45: error: reference
to ‘math’ is ambiguous
/users/zhlou/usr/include/armadillo_bits/constants_compat.hpp:158: error:
candidates are: typedef class arma::Math<double> arma::math
/users/zhlou/mlpack-sa/src/mlpack/../mlpack/core/math/clamp.hpp:14: error:
                namespace mlpack::math { }

The compilation is happened on a CentOS 6 with gcc 4.4.7 and my custom
compiled boost and armadillo so this may not be replicated on other
configuration. My workaround is to specify mlpack::math for all 4 cases
inside svd_test.cpp.

Best regards,

Zhihao Lou
PhD Candidate
Department of Computer Science
The University of Chicago


On Thu, Jul 3, 2014 at 1:10 PM, Zhihao Lou <lzh1984 at gmail.com> wrote:

> Ahh... You caught me right before I'm set for the independence day break.
> I'll take a look at the code next week.
>
> And yes it's my honor to be listed as one of the contributors.
>
> Now onto the search space. I can think of two ways of working with bounded
> search space.
>
> One is to let the function being optimized simply return something like
> DBL_MAX (or numeric_limits<double>::max() if you prefer c++ style) when any
> parameter go outside of the search space. This way there is no change
> needed for the annealing code. I've used this method in one of my problems
> and I can report that at least in my problem, I haven't seen any instances
> that repeatedly hit the boundary. In addition, it will allow the user to
> switch between hard and soft bound (like a penalty term) with relatively
> less changes.
>
> The other way is to let the function somehow report its search space,
> likely via a function call that returns 2 matrices, lowerBound and
> upperBound, each has the same dimension as the function being
> optimized.  I'm not sure how such boundary information is passed in other
> optimizers. But once that is done, the move generation can cut off at the
> boundary and alter the move distribution to one sided exponential while the
> parameter in question is at the boundary.
>
> Let me know your thoughts.
>
> Best regards,
>
> Zhihao Lou
> PhD Candidate
> Department of Computer Science
> The University of Chicago
>
>
> On Thu, Jul 3, 2014 at 12:14 PM, Ryan Curtin <gth671b at mail.gatech.edu>
> wrote:
>
>> On Fri, Apr 04, 2014 at 05:56:00PM -0400, Ryan Curtin wrote:
>> > On Tue, Mar 25, 2014 at 04:06:12PM -0500, Zhihao Lou wrote:
>> > > Hi Ryan,
>> > >
>> > > I've got the  SA done. It currently can only handle unbounded search
>> space,
>> > > but otherwise it should work. Where should I check if the search
>> space has
>> > > hard boundary?
>>
>> Ack, sorry to bombard you with multiple emails in a day, but I realized
>> I had not said anything about a bounded search space.  I don't have a
>> quick solution for this.  Do you have any ideas?  I'm sure we can figure
>> something out, if that is support that you would use if it was added.
>>
>> Also, may I include your name/email/website on the list of contributors
>> both in the code and on the website?
>>
>> Thanks,
>>
>> Ryan
>>
>> --
>> Ryan Curtin    | "I'm just going to shoot you once!"
>> ryan at ratml.org |   - Joseph Dunn
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cc.gatech.edu/pipermail/mlpack/attachments/20140707/6bed503a/attachment-0003.html>
-------------- next part --------------
Index: src/mlpack/core/optimizers/sa/sa_impl.hpp
===================================================================
--- src/mlpack/core/optimizers/sa/sa_impl.hpp	(revision 16773)
+++ src/mlpack/core/optimizers/sa/sa_impl.hpp	(working copy)
@@ -59,7 +59,7 @@
 
   size_t frozenCount = 0;
   double energy = function.Evaluate(iterate);
-  size_t oldEnergy = energy;
+  double oldEnergy = energy;
   math::RandomSeed(std::time(NULL));
 
   size_t idx = 0;
@@ -87,7 +87,7 @@
       frozenCount = 0;
 
     // Terminate, if possible.
-    if (frozenCount >= maxToleranceSweep * iterate.n_elem)
+    if (frozenCount >= maxToleranceSweep * moveCtrlSweep * iterate.n_elem)
     {
       Log::Debug << "SA: minimized within tolerance " << tolerance << " for "
           << maxToleranceSweep << " sweeps after " << i << " iterations; "