[mlpack] Cross-validation and hyper-parameter tuning infrastructure

[Kirill Mishchenko]

Hi Ryan.

By now it is hard for me to imagine how to make grid search optimiser to have a similar interface to already implemented optimisers like SGD since they work in slightly different domains. I guess a reasonable interface for grid search optimiser will allow such usage.

  arma::mat data /* = ... */;
  arma::Row<size_t> labels /* = ... */;

  GridSearchOptimizer<SoftmaxRegression<>, Accuracy, KFoldCV>
      softmaxOptimizer(data, labels);

  std::array<size_t, 1> numClasses = {5};
  arma::vec lambdas = arma::logspace(-3, 1); // {0.001, 0.01, 0.1, 1}

  std::tuple<size_t, double> bestSoftmaxParams =
      softmaxOptimizer.Optimize(numClasses, lambdas);
  double bestSoftmaxAccuracy = softmaxOptimizer.BestMeasurement();
  SoftmaxRegression<>& bestSoftmaxModel = softmaxOptimizer.BestModel();

Here when we call Optimize we pass a collection of possible values for each hyper-parameter (each additional constructor argument) of SoftmaxRegression<>. A similar interface of grid search for hyper-parameters is provided by scikit-learn: http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html <http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html>. The result of the Optimize call is a tuple of selected values (one per hyper-parameter) - I think it is good to return it here since only here we can deduct what type the tuple should have (if we want to be able to provide the tuple of selected hyper-parameters by another call after optimizing, we need to pass information about the tuple type when constructing a GridSearchOptimizer object).

On the other hand, SGT has the following signature for Optimize:

  double Optimize(arma::mat& iterate);

which I think can’t be easily utilized by grid search. Maybe you have some ideas?

Best regards,

Kirill Mishchenko


> On 4 Apr 2017, at 22:28, Ryan Curtin <ryan at ratml.org> wrote:
> 
> On Sat, Apr 01, 2017 at 10:55:45AM +0500, Kirill Mishchenko wrote:
>> Hi Ryan.
>> 
>> I’m planning to implement the following functionality as a GSoC project:
>> Measurements
>> Accuracy
>> Mean squared error
>> Precision
>> Recall
>> F1
>> Validation
>> Simple validation (splitting data once with validation set size specified by a user)
>> K-fold cross validation
>> Hyper-parameter tuning
>> Grid search based tuning
>> 
>> Does it seem as a reasonable set of functionality? I have decided to
>> include simple validation since it can be more appropriate when we
>> have a lot of training data or when training is a time consuming
>> process.
> 
> Hi Kirill,
> 
> That sounds good to me.  It would be nice if the grid search optimizer
> could have a similar API to the mlpack optimizers, so in that way,
> possibly other optimizers could be used.  (i.e. you could use SGD to
> find the best value of some continuous parameters.  This strategy falls
> apart a little bit when the hyperparameters are not of continuous type,
> so clearly we'll need to extend the OptimizerType paradigm at least a
> little bit.  But we can figure that out when we get there.)
> 
> Thanks,
> 
> Ryan
> 
> -- 
> Ryan Curtin    | "Do I sound like I'm ordering a pizza?"
> ryan at ratml.org <mailto:ryan at ratml.org> |   - John McClane

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20170407/26db1b60/attachment.html>