Neural Architecture Transfer

Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose Neural Architecture Transfer (NAT) to overcome this limitation. NAT is designed to efficiently generate task-specific custom models that are competitive under multiple conflicting objectives. To realize this goal we learn task-specific supernets from which specialized subnets can be sampled without any additional training. The key to our approach is an integrated online transfer learning and many-objective evolutionary search procedure. A pre-trained supernet is iteratively adapted while simultaneously searching for task-specific subnets. We demonstrate the efficacy of NAT on 11 benchmark image classification tasks ranging from large-scale multi-class to small-scale fine-grained datasets. In all cases, including ImageNet, NATNets improve upon the state-of-the-art under mobile settings ($\leq$ 600M Multiply-Adds). Surprisingly, small-scale fine-grained datasets benefit the most from NAT. At the same time, the architecture search and transfer is orders of magnitude more efficient than existing NAS methods. Overall, the experimental evaluation indicates that, across diverse image classification tasks and computational objectives, NAT is an appreciably more effective alternative to conventional transfer learning of fine-tuning weights of an existing network architecture learned on standard datasets. Code is available at https://github.com/human-analysis/neural-architecture-transfer

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Neural Architecture Search CIFAR-10 NAT-M3 Top-1 Error Rate 1.8% # 3
Search Time (GPU days) 1.0 # 18
Parameters 6.2M # 38
FLOPS 392M # 34
Neural Architecture Search CIFAR-10 NAT-M4 Top-1 Error Rate 1.6% # 1
Search Time (GPU days) 1.0 # 18
Parameters 6.9M # 39
FLOPS 468M # 35
Image Classification CIFAR-10 NAT-M4 Percentage correct 98.4 # 33
PARAMS 6.9M # 199
Image Classification CIFAR-10 NAT-M3 Percentage correct 98.2 # 41
PARAMS 6.2M # 198
Image Classification CIFAR-10 NAT-M2 Percentage correct 97.9 # 54
PARAMS 4.6M # 195
Image Classification CIFAR-10 NAT-M1 Percentage correct 97.4 # 76
PARAMS 4.3M # 193
Neural Architecture Search CIFAR-10 NAT-M1 Top-1 Error Rate 2.6% # 28
Search Time (GPU days) 1.0 # 18
Parameters 4.3M # 35
FLOPS 232M # 31
Neural Architecture Search CIFAR-10 NAT-M2 Top-1 Error Rate 2.1% # 8
Search Time (GPU days) 1.0 # 18
Parameters 4.6M # 36
FLOPS 291M # 32
Image Classification CIFAR-100 NAT-M3 Percentage correct 87.7 # 42
PARAMS 7.8M # 187
Neural Architecture Search CIFAR-100 NAT-M4 FLOPS 796M # 12
Percentage Error 11.7 # 1
PARAMS 9.0M # 11
Neural Architecture Search CIFAR-100 NAT-M3 FLOPS 492M # 11
Percentage Error 12.3 # 3
PARAMS 7.8M # 10
Neural Architecture Search CIFAR-100 NAT-M2 FLOPS 398M # 9
Percentage Error 12.5 # 4
PARAMS 6.4M # 9
Neural Architecture Search CIFAR-100 NAT-M1 FLOPS 261M # 8
Percentage Error 14.0 # 6
PARAMS 3.8M # 8
Image Classification CIFAR-100 NAT-M4 Percentage correct 88.3 # 39
PARAMS 9.0M # 188
Image Classification CIFAR-100 NAT-M1 Percentage correct 86.0 # 57
PARAMS 3.8M # 185
Image Classification CIFAR-100 NAT-M2 Percentage correct 87.5 # 44
PARAMS 6.4M # 186
Neural Architecture Search CIFAR-10 Image Classification NAT-M2 Percentage error 2.1 # 7
Params 4.6M # 11
FLOPS 291M # 16
Neural Architecture Search CIFAR-10 Image Classification NAT-M3 Percentage error 1.8 # 2
Params 6.2M # 14
FLOPS 392M # 17
Neural Architecture Search CIFAR-10 Image Classification NAT-M4 Percentage error 1.6 # 1
Params 6.9M # 15
FLOPS 468M # 18
Neural Architecture Search CIFAR-10 Image Classification NAT-M1 Percentage error 2.6 # 15
Params 4.3M # 10
FLOPS 232M # 15
Image Classification CINIC-10 NAT-M2 Accuracy 94.1 # 4
FLOPS 411M # 8
PARAMS 6.2M # 8
Neural Architecture Search CINIC-10 NAT-M4 Accuracy (%) 94.8 # 1
FLOPS 710M # 4
PARAMS 9.1M # 4
Neural Architecture Search CINIC-10 NAT-M3 Accuracy (%) 94.3 # 2
FLOPS 501M # 3
PARAMS 8.1M # 3
Neural Architecture Search CINIC-10 NAT-M2 Accuracy (%) 94.1 # 3
FLOPS 411M # 2
PARAMS 6.2M # 2
Image Classification CINIC-10 NAT-M3 Accuracy 94.3 # 3
FLOPS 501M # 9
PARAMS 8.1M # 9
Neural Architecture Search CINIC-10 NAT-M1 Accuracy (%) 93.4 # 4
FLOPS 317M # 1
PARAMS 4.6M # 1
Image Classification CINIC-10 NAT-M1 Accuracy 93.4 # 5
FLOPS 317M # 7
PARAMS 4.6M # 7
Neural Architecture Search DTD NAT-M4 Accuracy (%) 79.1 # 1
FLOPS 560M # 4
PARAMS 6.3M # 1
Neural Architecture Search DTD NAT-M1 Accuracy (%) 76.1 # 4
FLOPS 136M # 1
PARAMS 2.2M # 4
Neural Architecture Search DTD NAT-M2 Accuracy (%) 77.6 # 3
FLOPS 297M # 2
PARAMS 4.0M # 3
Neural Architecture Search DTD NAT-M3 Accuracy (%) 78.4 # 2
FLOPS 347M # 3
PARAMS 4.1M # 2
Neural Architecture Search FGVC Aircraft NAT-M1 Accuracy (%) 87.0 # 4
FLOPS 175M # 1
PARAMS 3.2M # 1
Neural Architecture Search FGVC Aircraft NAT-M3 Accuracy (%) 90.1 # 2
FLOPS 388M # 3
PARAMS 5.1M # 3
Fine-Grained Image Classification FGVC Aircraft NAT-M3 Accuracy 90.1% # 45
FLOPS 388M # 53
PARAMS 5.1M # 53
Fine-Grained Image Classification FGVC Aircraft NAT-M1 Accuracy 87.0% # 49
FLOPS 175M # 51
PARAMS 3.2M # 51
Neural Architecture Search FGVC Aircraft NAT-M4 Accuracy (%) 90.8 # 1
FLOPS 581M # 4
PARAMS 5.3M # 4
Fine-Grained Image Classification FGVC Aircraft NAT-M2 Accuracy 89.0% # 47
FLOPS 235M # 52
PARAMS 3.4M # 52
Fine-Grained Image Classification FGVC Aircraft NAT-M4 Accuracy 90.8% # 44
FLOPS 581M # 54
PARAMS 5.3M # 54
Neural Architecture Search FGVC Aircraft NAT-M2 Accuracy (%) 89.0 # 3
FLOPS 235M # 2
PARAMS 3.4M # 2
Image Classification Flowers-102 NAT-M4 Accuracy 98.3% # 24
FLOPS 400M # 50
PARAMS 4.2M # 47
Image Classification Flowers-102 NAT-M2 Accuracy 97.9% # 29
FLOPS 195M # 48
PARAMS 3.4M # 45
Image Classification Flowers-102 NAT-M3 Accuracy 98.1% # 28
FLOPS 250M # 49
PARAMS 3.7M # 46
Image Classification Flowers-102 NAT-M1 FLOPS 152M # 47
PARAMS 3.3M # 44
Neural Architecture Search Food-101 NAT-M3 Accuracy (%) 89.0 # 2
FLOPS 299M # 3
PARAMS 3.9M # 2
Fine-Grained Image Classification Food-101 NAT-M2 Accuracy 88.5 # 13
FLOPS 266M # 13
PARAMS 4.1M # 14
Neural Architecture Search Food-101 NAT-M1 Accuracy (%) 87.4 # 4
FLOPS 198M # 1
PARAMS 3.1M # 1
Fine-Grained Image Classification Food-101 NAT-M1 Accuracy 87.4 # 14
FLOPS 198M # 12
PARAMS 3.1M # 12
Neural Architecture Search Food-101 NAT-M2 Accuracy (%) 88.5 # 3
FLOPS 266M # 2
PARAMS 4.1M # 3
Fine-Grained Image Classification Food-101 NAT-M4 Accuracy 89.4 # 11
FLOPS 361M # 15
PARAMS 4.5M # 15
Fine-Grained Image Classification Food-101 NAT-M3 Accuracy 89.0 # 12
FLOPS 299M # 14
PARAMS 3.9M # 13
Neural Architecture Search Food-101 NAT-M4 Accuracy (%) 89.4 # 1
FLOPS 361M # 4
PARAMS 4.5M # 4
Neural Architecture Search ImageNet NAT-M2 Top-1 Error Rate 21.4 # 44
Accuracy 78.6 # 35
Params 7.7M # 13
MACs 312M # 90
Neural Architecture Search ImageNet NAT-M1 Top-1 Error Rate 22.5 # 62
Accuracy 77.5 # 50
Params 6.0M # 23
MACs 225M # 75
Neural Architecture Search ImageNet NAT-M3 Top-1 Error Rate 20.1 # 24
Accuracy 79.9 # 19
Params 9.1M # 8
MACs 490M # 117
Image Classification ImageNet NAT-M4 Top 1 Accuracy 80.5% # 638
Number of params 9.1M # 467
Hardware Burden None # 1
Operations per network pass None # 1
Neural Architecture Search ImageNet NAT-M4 Top-1 Error Rate 19.5 # 15
Accuracy 80.5 # 11
Params 9.1M # 8
MACs 600M # 131
Neural Architecture Search Oxford 102 Flowers NAT-M3 Accuracy (%) 98.1 # 2
FLOPS 250M # 3
PARAMS 3.7M # 3
Neural Architecture Search Oxford 102 Flowers NAT-M4 Accuracy (%) 98.3 # 1
FLOPS 400M # 4
PARAMS 4.2M # 4
Fine-Grained Image Classification Oxford 102 Flowers NAT-M2 Accuracy 97.9% # 16
FLOPS 195M # 24
PARAMS 3.4M # 20
Fine-Grained Image Classification Oxford 102 Flowers NAT-M3 Accuracy 98.1% # 15
FLOPS 250M # 25
PARAMS 3.7M # 21
Fine-Grained Image Classification Oxford 102 Flowers NAT-M4 Accuracy 98.3% # 13
FLOPS 400M # 26
PARAMS 4.2M # 22
Fine-Grained Image Classification Oxford 102 Flowers NAT-M1 FLOPS 152M # 23
PARAMS 3.3M # 19
Neural Architecture Search Oxford 102 Flowers NAT-M2 Accuracy (%) 97.9 # 3
FLOPS 195M # 2
PARAMS 3.4M # 2
Neural Architecture Search Oxford 102 Flowers NAT-M1 Accuracy (%) 97.5 # 4
FLOPS 152M # 1
PARAMS 3.3M # 1
Fine-Grained Image Classification Oxford-IIIT Pet Dataset NAT-M1 FLOPS 160M # 14
PARAMS 4.0M # 13
Neural Architecture Search Oxford-IIIT Pet Dataset NAT-M4 Accuracy (%) 94.3 # 1
FLOPS 744M # 4
PARAMS 8.5M # 4
Neural Architecture Search Oxford-IIIT Pet Dataset NAT-M2 Accuracy (%) 93.5 # 3
FLOPS 306M # 2
PARAMS 5.5M # 2
Neural Architecture Search Oxford-IIIT Pet Dataset NAT-M1 Accuracy (%) 91.8 # 4
FLOPS 160M # 1
PARAMS 4.0M # 1
Neural Architecture Search Oxford-IIIT Pet Dataset NAT-M3 Accuracy (%) 94.1 # 2
FLOPS 471M # 3
PARAMS 5.7M # 3
Fine-Grained Image Classification Oxford-IIIT Pet Dataset NAT-M3 Top-1 Error Rate 5.9% # 3
Accuracy 94.1% # 13
FLOPS 471M # 16
PARAMS 5.7M # 15
Fine-Grained Image Classification Oxford-IIIT Pet Dataset NAT-M2 Top-1 Error Rate 6.5% # 4
Accuracy 93.5% # 14
FLOPS 306M # 15
PARAMS 5.5M # 14
Fine-Grained Image Classification Oxford-IIIT Pet Dataset NAT-M4 Top-1 Error Rate 5.7% # 2
Accuracy 94.3% # 12
FLOPS 744M # 17
PARAMS 8.5M # 16
Neural Architecture Search Stanford Cars NAT-M3 Accuracy (%) 92.6 # 2
FLOPS 289M # 3
PARAMS 3.5M # 3
Neural Architecture Search Stanford Cars NAT-M2 Accuracy (%) 92.2 # 3
FLOPS 222M # 2
PARAMS 2.7M # 2
Neural Architecture Search Stanford Cars NAT-M1 Accuracy (%) 90.0 # 4
FLOPS 165M # 1
PARAMS 2.4M # 1
Fine-Grained Image Classification Stanford Cars NAT-M4 Accuracy 92.9% # 60
FLOPS 369M # 73
PARAMS 3.7M # 71
Fine-Grained Image Classification Stanford Cars NAT-M3 Accuracy 92.6% # 64
FLOPS 289M # 72
PARAMS 3.5M # 70
Neural Architecture Search Stanford Cars NAT-M4 Accuracy (%) 92.9 # 1
FLOPS 369M # 4
PARAMS 3.7M # 4
Fine-Grained Image Classification Stanford Cars NAT-M2 Accuracy 92.2% # 67
FLOPS 222M # 71
PARAMS 2.7M # 69
Fine-Grained Image Classification Stanford Cars NAT-M1 Accuracy 90.9% # 68
FLOPS 165M # 70
PARAMS 2.4M # 68
Image Classification STL-10 NAT-M4 Percentage correct 97.9 # 11
FLOPS 573M # 118
PARAMS 7.5M # 116
Neural Architecture Search STL-10 NAT-M4 Accuracy (%) 97.9 # 1
FLOPS 573M # 4
PARAMS 7.5M # 3
Neural Architecture Search STL-10 NAT-M1 Accuracy (%) 96.7 # 4
FLOPS 240M # 1
PARAMS 4.4M # 1
Neural Architecture Search STL-10 NAT-M2 Accuracy (%) 97.2 # 3
FLOPS 303M # 2
PARAMS 5.1M # 2
Neural Architecture Search STL-10 NAT-M3 Accuracy (%) 97.8 # 2
FLOPS 436M # 3
PARAMS 7.5M # 3
Image Classification STL-10 NAT-M2 Percentage correct 97.2 # 14
FLOPS 303M # 116
PARAMS 5.1M # 115
Image Classification STL-10 NAT-M3 Percentage correct 97.8 # 12
FLOPS 436M # 117
PARAMS 7.5M # 116
Image Classification STL-10 NAT-M1 Percentage correct 96.7 # 16
FLOPS 240M # 115
PARAMS 4.4M # 114

Methods


No methods listed for this paper. Add relevant methods here