Keywords: machine learning, hyperparameter optimization, tuning, classification, networked science
Webpages:
https://jakob-r.github.io/mlrHyperopt/ Most machine learning tasks demand hyperparameter tuning to achieve a good performance. For example, Support Vector Machines with radial basis functions are very sensitive to the choice of both kernel width and soft margin penalty C. However, for a wide range of machine learning algorithms these “search spaces” are less known. Even worse, experts for the particular methods might have conflicting views. The popular package
caret (Jed Wing et al. 2016) approaches this problem by providing two simple optimizers
grid search and
random search and individual search spaces for all implemented methods. To prevent training on misconfigured methods a
grid search is performed by default. Unfortunately it is only documented which parameters will be tuned but the exact bounds have to be obtained from the source code. As a counterpart
mlr (Bischl et al. 2016) offers more flexible parameter tuning methods such as an interface to
mlrMBO (Bischl et al. 2017) for conducting Bayesian optimization. Unfortunately
mlr lacks of default search spaces and thus parameter tuning becomes difficult. Here
mlrHyperopt steps in to make hyperparameter optimization as easy as in
caret. As a matter of fact, for a developer of a machine learning package, it is unquestionable impossible to be an expert of all implemented methods and provide perfect search spaces. Hence
mlrHyperopt aims at:
- improving the search spaces of caret with simple tricks.
- letting the users submit and download improved search spaces to a database.
- providing advanced tuning methods interfacing mlr and mlrMBO.
A study on selected data sets and numerous popular machine learning methods compares the performance of the grid and random search implemented in
caret to the performance of
mlrHyperopt for different budgets.
References Bischl, Bernd, Michel Lang, Lars Kotthoff, Julia Schiffner, Jakob Richter, Erich Studerus, Giuseppe Casalicchio, and Zachary M. Jones. 2016. “Mlr: Machine Learning in R.”
Journal of Machine Learning Research 17 (170): 1–5.
https://CRAN.R-project.org/package=mlr.
Bischl, Bernd, Jakob Richter, Jakob Bossek, Daniel Horn, Janek Thomas, and Michel Lang. 2017. “mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions.”
arXiv:1703.03373 [Stat], March.
http://arxiv.org/abs/1703.03373.
Jed Wing, Max Kuhn. Contributions from, Steve Weston, Andre Williams, Chris Keefer, Allan Engelhardt, Tony Cooper, Zachary Mayer, et al. 2016.
Caret: Classification and Regression Training.
https://CRAN.R-project.org/package=caret.
