Comparative Study of Surrogate Techniques for CNN Hyperparameter Optimization

Optimizing hyper parameters in Convolutional Neural networks is a tedious process for many researchers and practitioners. It requires a high degree of expertise or experience to optimise the hyper parameters, and manual optimisation is likely to be biased. To date, methods or approaches to automate hyper parameter optimization include grid search, random search, and Genetic Algorithms (GAs). However, evaluating large number of sample points in the hyperparameter configuration space, as is typically required by these methods, is computationally expensive process. Hence, the objective of this paper is to explore regression as a surrogate technique in CNN hyperparameter optimisation. Performance in terms of accuracy, error rate, training time and coefficient of determination (R2) are evaluated and recorded. Although there is no significant performance difference between the resulting optimized Deep Learning and state-of-the-art on CIFAR-10 datasets, using
regression as a surrogate technique for CNN hyperparameter optimization contributes to minimising the time taken for the optimization process, a benefit which has not been fully explored in the literature to the best of the author’s knowledge.