Reducing Regression Time in Neural Network using Nth Absolute Root Mean Error
Siddhartha Dhar Choudhury1, Shashank Pandey2, Kunal Mehrotra3, Christhu Raj4, Rajeev S5
1Siddhartha Dhar Choudhury, SRMIST, Indian Institute of Technology Madras, Chennai (Tamil Nadu), India.
2Shashank Pandey, SRMIST, Indian Institute of Technology Madras, Chennai (Tamil Nadu), India.
3Kunal Mehrotra, SRMIST, Indian Institute of Technology Madras, Chennai (Tamil Nadu), India.
4Christhu Raj, SRMIST, Indian Institute of Technology Madras, Chennai (Tamil Nadu), India.
5Rajeev S,SRMIST, Indian Institute of Technology Madras, Chennai (Tamil Nadu), India.
Manuscript received on October 15, 2019. | Revised Manuscript received on 24 October, 2019. | Manuscript published on November 10, 2019. | PP: 4981-4995 | Volume-9 Issue-1, November 2019. | Retrieval Number: J96260881019/2019©BEIESP | DOI: 10.35940/ijitee.A3912.119119
Open Access | Ethics and Policies | Cite | Mendeley | Indexing and Abstracting
© The Authors. Blue Eyes Intelligence Engineering and Sciences Publication (BEIESP). This is an open access article under the CC-BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Abstract: Neural networks require a large dataset to learn features, and this takes high computational power, even using high end Graphical Processing Units (GPUs) some use cases might require days of continuous training. This can be solved either by making hardware specific changes to code or by replacing the algorithm used. In this paper we propose a novel loss function called Nth Absolute Root Mean Error (NARME) which speeds up the training process for a set of supervised learning problems, specifically regression (which is a problem in which we predict value of a continuous variable). In this function, we take nth root of the absolute difference between the predicted and the actual output, then calculate the mean over the entire set of data points available in the dataset, thus in a way starting at a much lower point in the gradient descent curve than other commonly used regression losses. The loss function (NARME) places the value of loss after the first iteration to a point that is closer to the minima (which is the goal of the training process), and then takes small number of descent steps in the n-dimensional space to reach the global minima. NARME has been found to work with commonly used neural network architectures and is shown to work exceptionally well with Neural Arithmetic and Logic Unit cell type networks. The usage of this type loss function in regression problems helps speed the training process by a factor of ten in most of the cases, due to the small number of steps it requires to reach the minima.
Keywords: Loss Function, Regression, Neural Network, Training Time.
Scope of the Article: Regression and Prediction