[mlpack] GSoC 2018: Particle Swarm Optimization

[Chintan Soni]

Hello Marcus,

I went through the PR you mentioned a bit more thoroughly; I had
missed the use of the velocity update policy. It's probably the best
way to incorporate multiple variants of the algorithm.

>> Looks really interesting, the "Better initialization" point is definitely something that might affect the overall training time in a positive way.

Just had an idea regarding this, it involves slightly modifying the
way iterations are run as follows:

1. Randomly distribute particles over the search space (standard
initialization method)

2. User specifies an initial point, currently what f.GetInitialPoint()
does for the test functions. This is usually the weight matrix saved
from a run of a training algorithm.

3. We run 1 (or a very small number of) iteration(s) on the randomly
distributed particles and evaluate the fitness values of each
particle.

4. Pick the worst particle from the swarm and replace it with the user
provided initialization point (if it is better than this "worst", else
discard it completely).

The above steps ensure that each run of the algorithm makes use of the
user provided initial point, and benefits from the random distribution
as well. This method can also be extended to provide a better initial
point for gradient descent:

5. Run the standard PSO algorithm for x% of the max iterations, and at
the end of the x% iterations, evaluate the best particle. This
particle will serve as the initialization point for the gradient
descent step.

6. Run GD for the next (100-x)% iterations.

It goes without saying that the algorithm will return as soon as the
fitness reaches the tolerance threshold.

The value of x depends mostly on the objective function complexity; if
the objective function is computationally straining to evaluate, a
smaller value of x will favour gradient descent over PSO and still
influence training time in a positive way, whereas if the objective
function is relatively simpler, PSO might converge faster. It might
help to make x configurable as a user input.

Regarding the time and resources being consumed by the build process,
I have disabled tests entirely for the time being and linked against
the generated library with external code; this has brought down the
build time down to a few minutes and is much more resource friendly.

I have set up a basic layout with some files (imitating the structure
of other optimizers) and started coding; it might take some time to
have a basic working PSO as I only have a few hours each weekday and
mostly get work done over the weekend. I will submit a PR as soon as
possible.

Another doubt: the application guide mentions adding a link to
"Homepage", is it necessary to have one?

Please let me know your thoughts about the approach mentioned above.
Will keep you posted about my progress.

Thanks and regards,
Chintan