[mlpack] GSoC 2018: Particle Swarm Optimization

[Marcus Edel]

Hello Chintan,

thanks for the update.

> The above steps ensure that each run of the algorithm makes use of the
> user provided initial point, and benefits from the random distribution
> as well. This method can also be extended to provide a better initial
> point for gradient descent:

Really like the ideas, and I think this is something that could be explored,
might be a neat research component.

> I have set up a basic layout with some files (imitating the structure
> of other optimizers) and started coding; it might take some time to
> have a basic working PSO as I only have a few hours each weekday and
> mostly get work done over the weekend. I will submit a PR as soon as
> possible.

Sounds good, don't feel obligated to work on the parts before GSoC has started,
but it might be a good opportunity to get familiar with the codebase, your call.

> Another doubt: the application guide mentions adding a link to
> "Homepage", is it necessary to have one?

No, if you have one, you could add a reference, if not that's fine, no worries
this are optional informations.

Thanks,
Marcus

> On 14. Mar 2018, at 05:09, Chintan Soni <chintan.soni4 at gmail.com> wrote:
> 
> Hello Marcus,
> 
> I went through the PR you mentioned a bit more thoroughly; I had
> missed the use of the velocity update policy. It's probably the best
> way to incorporate multiple variants of the algorithm.
> 
>>> Looks really interesting, the "Better initialization" point is definitely something that might affect the overall training time in a positive way.
> 
> Just had an idea regarding this, it involves slightly modifying the
> way iterations are run as follows:
> 
> 1. Randomly distribute particles over the search space (standard
> initialization method)
> 
> 2. User specifies an initial point, currently what f.GetInitialPoint()
> does for the test functions. This is usually the weight matrix saved
> from a run of a training algorithm.
> 
> 3. We run 1 (or a very small number of) iteration(s) on the randomly
> distributed particles and evaluate the fitness values of each
> particle.
> 
> 4. Pick the worst particle from the swarm and replace it with the user
> provided initialization point (if it is better than this "worst", else
> discard it completely).
> 
> The above steps ensure that each run of the algorithm makes use of the
> user provided initial point, and benefits from the random distribution
> as well. This method can also be extended to provide a better initial
> point for gradient descent:
> 
> 5. Run the standard PSO algorithm for x% of the max iterations, and at
> the end of the x% iterations, evaluate the best particle. This
> particle will serve as the initialization point for the gradient
> descent step.
> 
> 6. Run GD for the next (100-x)% iterations.
> 
> It goes without saying that the algorithm will return as soon as the
> fitness reaches the tolerance threshold.
> 
> The value of x depends mostly on the objective function complexity; if
> the objective function is computationally straining to evaluate, a
> smaller value of x will favour gradient descent over PSO and still
> influence training time in a positive way, whereas if the objective
> function is relatively simpler, PSO might converge faster. It might
> help to make x configurable as a user input.
> 
> Regarding the time and resources being consumed by the build process,
> I have disabled tests entirely for the time being and linked against
> the generated library with external code; this has brought down the
> build time down to a few minutes and is much more resource friendly.
> 
> I have set up a basic layout with some files (imitating the structure
> of other optimizers) and started coding; it might take some time to
> have a basic working PSO as I only have a few hours each weekday and
> mostly get work done over the weekend. I will submit a PR as soon as
> possible.
> 
> Another doubt: the application guide mentions adding a link to
> "Homepage", is it necessary to have one?
> 
> Please let me know your thoughts about the approach mentioned above.
> Will keep you posted about my progress.
> 
> Thanks and regards,
> Chintan