Cross-Validation and Hyper-Parameter Tuning: Week 8

During the eighth week I was finishing working on the hyper-parameter tuning module. Some minor changes still should be added, but basically it is done. Since the pull request on the simple cross-validation strategy is not merged yet, I need to wait before sending a pull request for the hyper-parameter tuning module. By now it is available in my fork of mlpack. The same goes for k-fold cross-validation.

The main part of my GSoC plan is almost done, so I'm going to work on some optional parts. For the nearest time it will be additional metrics for the cross-validation and hyper-parameter tuning modules like precision, recall and F1.

more...

Cross-Validation and Hyper-Parameter Tuning: Week 7

During the seventh week I have been primarily working on implementation of different parts of the hyper-parameter tuning module. I have implemented the cross-validation wrapper that I mentioned previously, as well as grid-search based optimization. The grid-search optimizer follows the same interface as other mlpack optimizers, so it is possible to use it in other mlpack methods too.

Now I'm working on a hyper-parameter tuner itself. It should be able to accept different types of information for hyper-parameters - e.g. that some hyper-parameters should be bound to particular values, while others be chosen from sets of values.

I haven't sent any of the implemented parts for review yet since my latest PR is still under review (and the new code depends on this one). Once my latest PR is merged and the hyper-parameter tuner is ready, I will send the whole hyper-parameter tuning module as a PR.

more...

Cross-Validation and Hyper-Parameter Tuning: Week 6

During the sixth week I implemented k-fold cross-validation. I haven't sent it for review yet since the previous PR (#1044) is still under consideration, and the new code depends on this PR.

On the sixth week I also started to work on the hyper-parameter tuning module by implementing a wrapper for cross-validation strategies. This wrapper serves for adapting the interface of the cross-validation classes to the one that can be utilized by the mlpack optimizers. It also makes it possible to bind some arguments (hyper-parameters) to some specific values.

On the next week I plan to finish my work on the cross-validation wrapper and to start implementing the grid-search optimizer (iterating through points on a multidimensional grid).

more...

Cross-Validation and Hyper-Parameter Tuning: Week 5

During the fifth week I was primarily working on a cross-validation infrastructure and the simple cross-validation strategy. The main part of the work has been done, and now it is in the review stage (#1044).

The next task is to implement k-fold cross-validation strategy. Luckily, almost all secondary stuff is implemented, and coding can be focused on the strategy itself. The main challenge here is to avoid creating multiple copies of data while preparing different subsets.

more...

Cross-Validation and Hyper-Parameter Tuning: Week 4

During the fourth week I was working on a class that implements a simple cross-validation strategy. The strategy by itself is not difficult to implement, but supporting different types of machine learning algorithms is a bit tricky. It also would be nice to design a solution that could be re-used by other cross-validation strategies. It takes a bit more time than I initially planed (now I plan to finish it on this week), but hopefully a good design will save some time latter.

more...

Cross-Validation and Hyper-Parameter Tuning: Week 3

During the third week I was working on meta tools that for a given machine learning algorithm allow to extract meta information about it, e.g. predictions type, whether it takes a data::DatasetInfo parameter along with data and whether it supports weighted learning (#1031).

Now the developed tools can facilitate implementation of various cross validation strategies. I will start with a simple one - splitting data into training and validation sets in according with a given proportion.

more...

Cross-Validation and Hyper-Parameter Tuning: Week 2

During the second week I was working on tools that allow to check whether a given class has a method with particular name and form (#1020). A usage example can be found above the implementation of the macro 'HAS_METHOD_FORM`.

The method form detection tool will allow for a given machine learning algorithm to automatically extract information such as predictions type, whether it takes DatasetInfo parameter along with data and whether it supports weighted learning. We need to do it to prevent users from passing such information to prospective cross-validation/hyper-parameter tuning modules.

The next step is to facilitate the implemented macro HAS_METHOD_FORM for things mentioned above. I have already started to work on it and plan to send a PR this week.

more...

Cross-Validation and Hyper-Parameter Tuning: Community Bonding Period and Week 1

Before coding officially began I had done some preparatory work. Specifically, I had changed the interface for passing responses in LinearRegression and LARS (#1002). Now LinearRegression and LARS accept column-major responses as all other classification and regression algorithms in mlpack. The mentioned pull request also separates interfaces for weighted and non-weighted learning in LinearRegression, which will be handy for supporting LinearRegression in cross-validation and hyper-parameter tuning modules.

During the first week I implemented the accuracy and MSE metrics for cross-validation (#1016). Now I'm going to start working on cross-validation implementation that simply splits data into training and validation sets in according with a given proportion. In order to prevent users from passing redundant information about a machine learning algorithm (like whether it is a regression algorithm or a classification algorithm, or whether it supports weighted learning) and to extract this information automatically from the machine learning algorithm type, firstly I'm going to implement meta-programming tools that allow to extract such information.

more...