New AI Method Cuts Deep Learning Training up to 69 Percent

Source: geralt/Pixabay

Researchers at North Carolina State University have developed a method to reduce artificial intelligence (AI) training for deep learning by 69 percent and presented their paper at last month’s 35th IEEE International Conference on Data Engineering (ICDE 2019).

The research team of Dr. Xipeng Shen and PhD students Lin Ning and Hui Guan at North Carolina State University discovered a novel way, called Adaptive Deep Reuse (ADR), to reduce the computationally intensive and lengthy training process for Convolutional Neural Networks (CNNs), a class of AI deep neural networks that is frequently used for computer vision, without loss to precision.

The human brain and neuroscience have inspired concepts and methods used in artificial intelligence. Convolutional neural networks are somewhat analogous to the biological visual cortex, where cortical neurons fire in response to stimuli in the receptive field. Many recent advances in science and industry that involve data mining and engineering are due to convolutional neural networks.

The researchers based their approach on the observation that many pixels in images are similar in datasets. The approach is to find the groups of data that are similar in a dataset, then apply the results from filtering one group to all of the other groups with similar data. In the case of a dataset of images of beaches as an example, sand can be could be considered a group.

In this study, the researchers identified similarities existing among neuron vectors across the inputs of each convolutional layer using Locality Sensitive Hashing (LSH) as the clustering algorithm in the forward propagation, then reusing the similarity results in the backward propagation.

The team used different levels of precision tolerance through the different stages of training the convolutional neural network. The data group size was decreased and the similarity threshold tightened in later epochs. In machine learning, an epoch is when a whole data set is passed forward and backward only once through the neural network.

“Usually at early training iterations, since the model is very rough, the training of the model is hence less sensitive to approximation errors than in later stages,” wrote the researchers. “In later training stages when the model gets close to convergence, the model is well learned. A small change of the input matrix may lead to substantial errors in the model updates, causing the training slow to converge. Therefore, the basic idea of adaptive deep reuse is to be more aggressive on computation reuse in early stages and adjust the clustering parameters gradually so that we have less computation reuse but better precision in later stages.”

Using the Adaptive Deep Reuse approach, the team tested the performance on three common convolutional neural networks. The training time of the convolutional neural networks was reduced by 69 percent for AlexNet, 68 percent for VGG-19, and 63 percent for CifarNet, without impacting the precision and accuracy of the image classification.

The team wrote that their research was the “first study that systematically explores neuron vector similarities for speeding up CNN training,” and proves that backward propagation “could benefit directly from the neuron vector similarity detected in the forward propagation, which is the key point for efficient computation reuse in the backward propagation.”

Previous efforts to speed up the training of convolutional neural networks by other researchers centered on reducing computations of the convolutional layers and redundancies in weight parameters. This new research presents an alternative approach that not only significantly shortens the training time for deep learning, but also without any accuracy loss—paving the way for industry and science in the future.

Copyright © 2019 Cami Rosso All rights reserved.