Effects of Chosen Scalar Products on Gradient Descent Algorithms
The 28th International Conference of The Jangjeon Mathematical Society (ICJMS 2015), Antalya, Turkey, 15 - 19 May 2015, pp.90, (Full Text)
- Publication Type: Conference Paper / Full Text
- City: Antalya
- Country: Turkey
- Page Numbers: pp.90
- Akdeniz University Affiliated: Yes
Abstract
Steepest gradient descent algorithm, which is a line search method for finding a relative minimum or maximum of a function, is a simple and widely applied optimization technique. Steepest gradient descent is implemented without the need to solve or evaluate a Jacobian equation or associated linear system. The steepest descent algorithm can be improved by considering different aspects of a problem with the conjugate gradient technique. The choice of the metric, which the gradient depends on, notably affects the results of a gradient descent algorithm. In other words, the direction of the downhill depends on the chosen scalar product for the chosen function space. Careful design of the metric is crucial to reach better local minima of the function. Therefore, excellent convergence and smooth gradients can be obtained using different inner products. Generally, Euclidean type gradient (L²) is used to find the minimum or maximum point of a function. However, there are many undesired properties of the standard L² type gradient. For instance, there is nothing in the definition of the L² type inner product to discourage gradient flows in the space of curves when the flows are not smooth. Smooth but infinite dimensional function spaces provide several advantages. For instance, finding the minimum value of a function can be directly achieved without any restriction related to a finite dimensional space.