Analysis of the AutoML Challenge Series 2015–2018
The ChaLearn AutoML Challenge (The authors are in alphabetical order of last name, except the first author who did most of the writing and the second author who produced most of the numerical analyses and plots.) (NIPS 2015 – ICML 2016) consisted of six rounds of a machine learning competition of progressive difficulty, subject to limited computational resources. It was followed by a one-round AutoML challenge (PAKDD 2018). The AutoML setting differs from former model selection/hyper-parameter selection challenges, such as the one we previously organized for NIPS 2006: the participants aim to develop fully automated and computationally efficient systems, capable of being trained and tested without human intervention, with code submission. This chapter analyzes the results of these competitions and provides details about the datasets, which were not revealed to the participants. The solutions of the winners are systematically benchmarked over all datasets of all rounds and compared with canonical machine learning algorithms available in scikit-learn. All materials discussed in this chapter (data and code) have been made publicly available at http://automl.chalearn.org/.
PDF Abstract
Datasets
Introduced in the Paper:
Chalearn-AutoML-1
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| AutoML | Chalearn-AutoML-1 | marc.boulle | Rank (AutoML5) | 6.40 | # 8 | |
| Set1 (F1) | 0.7005 | # 7 | ||||
| Set2 (PAC) | 0.3698 | # 4 | ||||
| Set3 (AUC) | -1.0000 | # 8 | ||||
| Set4 (ABS) | 0.2507 | # 6 | ||||
| Set5 (BAC) | 0.4618 | # 7 | ||||
| Duration | 4603.81 | # 4 | ||||
| AutoML | Chalearn-AutoML-1 | reference_mb | Rank (AutoML5) | 5.20 | # 6 | |
| Set1 (F1) | 0.7005 | # 7 | ||||
| Set2 (PAC) | 0.3698 | # 4 | ||||
| Set3 (AUC) | 0.6776 | # 2 | ||||
| Set4 (ABS) | 0.2507 | # 6 | ||||
| Set5 (BAC) | 0.4618 | # 7 | ||||
| Duration | 4889.14 | # 5 | ||||
| AutoML | Chalearn-AutoML-1 | postech.mlg_exbrain | Rank (AutoML5) | 5.20 | # 6 | |
| Set1 (F1) | 0.7542 | # 4 | ||||
| Set2 (PAC) | 0.2802 | # 6 | ||||
| Set3 (AUC) | 0.3333 | # 5 | ||||
| Set4 (ABS) | 0.1507 | # 8 | ||||
| Set5 (BAC) | 0.5564 | # 5 | ||||
| Duration | 3343.64 | # 1 | ||||
| AutoML | Chalearn-AutoML-1 | abhishek4 | Rank (AutoML5) | 4.60 | # 5 | |
| Set1 (F1) | 0.7565 | # 2 | ||||
| Set2 (PAC) | 0.0172 | # 8 | ||||
| Set3 (AUC) | 0.2911 | # 7 | ||||
| Set4 (ABS) | 0.2791 | # 4 | ||||
| Set5 (BAC) | 0.5595 | # 3 | ||||
| Duration | 4353.45 | # 2 | ||||
| AutoML | Chalearn-AutoML-1 | djajetic | Rank (AutoML5) | 3.00 | # 2 | |
| Set1 (F1) | 0.7531 | # 5 | ||||
| Set2 (PAC) | 0.3905 | # 2 | ||||
| Set3 (AUC) | 0.6875 | # 1 | ||||
| Set4 (ABS) | 0.3067 | # 1 | ||||
| Set5 (BAC) | 0.5517 | # 6 | ||||
| Duration | 5842.12 | # 6 | ||||
| AutoML | Chalearn-AutoML-1 | aad_freiburg | Rank (AutoML5) | 1.60 | # 1 | |
| Set1 (F1) | 0.7947 | # 1 | ||||
| Set2 (PAC) | 0.4061 | # 1 | ||||
| Set3 (AUC) | 0.5543 | # 3 | ||||
| Set4 (ABS) | 0.2957 | # 2 | ||||
| Set5 (BAC) | 0.5900 | # 1 | ||||
| Duration | 5942.22 | # 8 | ||||
| AutoML | Chalearn-AutoML-1 | reference | Rank (AutoML5) | 4.40 | # 4 | |
| Set1 (F1) | 0.7556 | # 3 | ||||
| Set2 (PAC) | 0.0343 | # 7 | ||||
| Set3 (AUC) | 0.2927 | # 6 | ||||
| Set4 (ABS) | 0.2790 | # 5 | ||||
| Set5 (BAC) | 0.5601 | # 2 | ||||
| Duration | 4416.40 | # 3 | ||||
| AutoML | Chalearn-AutoML-1 | reference_ls | Rank (AutoML5) | 4.00 | # 3 | |
| Set1 (F1) | 0.7062 | # 6 | ||||
| Set2 (PAC) | 0.3708 | # 3 | ||||
| Set3 (AUC) | 0.5384 | # 4 | ||||
| Set4 (ABS) | 0.2856 | # 3 | ||||
| Set5 (BAC) | 0.5580 | # 4 | ||||
| Duration | 5879.88 | # 7 |
