Comparison Between Steepest Descent Method and Conjugate Gradient Method by Using Matlab

Abstract


Introduction
Optimization presents an essential tool in decision theory and analysis of physical systems.Optimization can be defined as the process of finding the best solution to a problem in a certain sense and under certain conditions [1].Optimization theory is a very developed area with its wide applications in science, engineering, business management, military and space technology, location science, statistics, portfolio analysis, and machine computations [1,2,3,4,5,6,7].Researchers have studied algorithms for minimizing continuously differentiable functions [1][2][3][4][5][6][7][8][9][10].
In particular, the Steepest descent algorithms have received much attention because of their simplicity and success in many applications.[8].The Steepest descent method (also known as Cauchy's or gradient method) is one of the earliest and most fundamental methods of unconstrained scalar optimization.[8,9,10].
The technique is fundamental from a theoretical viewpoint, by using a simple optimization algorithm, the method can find the local minimum of a function.Significant improvements in the functional performance of Steepest descent methods have been made over the last few decades merely by developing innovative approaches for selecting the step size in each iteration [8].Recently, different works on the Steepest descent method have been done [8,11,12,13,14].Fliege and Svaiter [15] presented a Pareto descent method for multi-objective optimizations.Drummond and Svaiter present a Cauchy-like method to solve smooth unconstrained vector optimization problems in [9].They have proved the global convergence for the weak unconstrained minimizer.Fliege and Svaiter have presented a Steepest descent technique for unconstrained multicriteria optimization and a method called "feasible descent direction" for restricted multicriteria optimization [16].
In the last decades, the interest in Gradient methods has been renewed after the innovative approach of Borwein and Barzilai [17], which stimulated novel choices for αk and proved to be largely superior to the Cauchy step length, they have derived two-point step sizes to a Steepest descent method by approximating the secant equations [17].Fletcher proposed a technique for quadratic and nonquadratic objective functions that takes advantage of the availability of a few more "long" vectors of storage to produce large performance improvements.[11].
The Conjugate gradient(CG) method is a method between the Newtons and Steepest descent method [18].The Conjugate gradient method diverts the Steepest descent method's direction by multiplying it by a positive multiple of the method's direction.[18].CG method is beneficial for minimizing functions of very many variables because it does not require the storage of any matrices.However, the algorithm's convergence rate is linear unless the iterative procedure is "restarted" occasionally [19].Fletcher and Reeves were the first to apply the Conjugate gradient algorithm to the general unconstrained minimization problem [20].
Fletcher and Reeves described a quadratically convergent gradient method for locating an unconstrained local minimum of a function of several variables.As a machine method, the Conjugate gradient method outperforms elimination.Because it gives a solution in n steps, it is more straightforward in coding and requires less storage space [21,22,23].Stanimirovic et al. investigated classify and compare main Quasi-Newton and Conjugate gradient methods to present a global overview of scientific advances in this field [24].Lasdon et al. are extended the Conjugate gradient minimization method of Fletcher and Reeves to optimal control problems.They have presented the Conjugate gradient method and compared the method with the method of Steepest descent.The convergence of the proposed method is much more rapid in all cases [25].
Optimization problems are extremely important when developing technological processes and projecting technical objects, as well as during their implementation and operation [26].Computer algorithms are effective processing methods.It is necessary to improve their efficiency and decrease their complexity in all implementation without loss of efficiency [27].Nonlinear programming in Matlab was used to solve optimization problems.[26,27,28,29,30,31].
Additionally, several fundamental algorithms are presented and analyzed to provide context for this critical component of optimization.MATLAB programs demonstrate many topics, and ideally, the interested reader will find satisfaction in the ability of actually solving problems [28].The reported case study  [32].Besides computational efficiency, another advantage of Matlab software is that their implementations for large-scale problems are considerably more straightforward than those for the other algorithms.This is particularly true in a high-level programming environment such as MATLAB, which supports high-level sparse-matrix operations [33,34].
Zhang used MATLAB software for solving Linear programs by the interior point method [33].MATLAB software was used to compare the Steepest descent method and the Conjugate gradient method in the presented work.

The Method of Steepest Descent
Let  have continuous first partial derivatives on   , we will frequently need the gradient vector of  and therefore, we introduce some simplifying notation.The gradient ∇() is, according to our conventions, defined as an n-dimensional row vector.For convenience, we define the n-dimensional column vector (  ) = ∇(  )
The Steepest descent method is defined by the following iterative algorithm where   is a nonnegative scalar minimizing (  −   ).In other words, from the point   we search along the direction of the negative gradient −  to a minimum point on this line; this minimum point is taken to be  +1 .

Algorithm 1. Steepest Descent Method
Step 1: Given an initial vector X0,  0 = ∇( 0 ) and a convergence tolerance e Step The hessian matrix is then we obtain Now, then Hence, ≮ , by repeating the same process, one can obtain a solution in 9 th iteration, since  9 = [0, Results are presented in table 2.

Conjugate Gradient Method
The Conjugate Gradient(CG) method is a fundamental algorithmic method for solving symmetric, positive definite systems of linear equations and minimizing unconstrained nonlinear functions.The Conjugate gradiet method is based on a simple procedure called the CG-standard for minimizing a strictly convex quadratic function [37,38] Where  is a positive definite symmetric  ×  matrix and  ∈ ℝ  is a known vector Observe that the Hessian of function ( 1) is independent of x.Since the Hessian Q is symmetric and positive definite, from the optimality conditions for a minimum of a differentiable function, it follows that there is a unique minimizer x* of (1).from (2), observes that x* is the solution of the linear system  = .
For any  0 ∈   the sequence {  } generated according to The algorithm of the Conjugate gradient method can be obtained as follow:

MATLAB COMPUTATIONS
We implemented the Matlab software for minimizing different functions.Algorithm 2 (which includes Algorithm 1 as a particular case) was implemented in Matlab software.This section describes the implementations of these algorithms and input parameter settings used in experiments [8]  By applying for the Matlab program, we will obtain the result.Using the Steepest descent method, (10) iterations are needed to minimize the given function, with an elapsed time of 0.094797 seconds.In comparison, the Conjugate gradient method requires (9) iterations with an elapsed time of 0.137276 seconds.Secondly, in the time aspects, by looking at Table 5, one can conclude that the Steepest descent method needs 0.083051seconds to do 32 iterations for minimizing a function (, ).On the other hand, the Conjugate gradient method needs 0.090517 seconds to do 14 iterations for finding min f.Similarly, for Example 2, the Steepest descent method minimized the function in 0.094797 seconds, while the Conjugate gradient method required 0.137276 seconds.We obtain that the Steepest descent method requires less time than the Conjugate gradient method to minimize the function.Finally, we conclude that the Conjugate gradient method is slower but more productive because it converges after fewer iterations.

Conclusions
In the present work, the Conjugate gradient method and the Steepest descent method have been studied.These methods are used for minimizing nonlinear functions.
|| 9 || < .The results are obtained in the Matlab program.The Steepest descent method minimized this function in 9 th iterations with an elapsed time of 0.074341 seconds.
Example 1 .Let (, ) =  2 +  2 +  − 3 , then find minimum point by the Steepest descent method  0 = (0,0)  and  = 0.01 Solution: By applying Algorithm 1, the gradient is  0 . For solving the nonlinear unconstrained optimization problem Where : ℝ  → ℝ is a continuously differentiable function, and any nonlinear Conjugate gradient algorithm generates the sequence {  }of the form  +1 =   +     where   is the stepsize obtained by line search and   is the search direction computed by  +1 = − +1 +     For  ≥ 0, where   is the Conjugate gradient parameter and  +1 = ∇(( +1 )) .In Conjugate gradient methods,  0 = − 0 .Nonlinear Conjugate gradient methods have a very nice theory, with a lot of important results on their convergence.The linear Conjugate gradient algorithm is dedicated to minimizing convex quadratic functions.Hestenes introduced this algorithm.Let us consider the quadratic function

Table 1
Results obtained for the problem in Example 1. by using the Conjugate gradient method For the same example, the Steepest descent method was applied.It is minimized in 9 iterations with an elapsed time of 0.074341 seconds.Obtained results are presented in Table2.
obtained that the min f solution needs only two iterations with an elapsed time of 0.077635 seconds, as shown in the Table1.

Table 2
Results obtained for the problem in Example 1. by using the Steepest descent method

Table 3
Results obtained for the problem in Example 2. by using the Conjugate gradient method

Table 4
Results obtained for the problem in Example 2. by using the Steepest descent method Using the Matlab program for the given function, we will obtain minimizing points that are different for both methods.The results are explained in the following table.

Table 5
In this table, the results of both methods are presented for Example 3.Generally, both methods are frequently used for minimizing functions.However, like other methods, there are many different advantages and disadvantages for techniques.We will compare both methods using the results of Example 2 and Example 3. the comparison has two aspects: time and efficiency.Firstly, in the case of efficiency analysis, by noticing table5.the Conjugate gradient method needs fewer iterations than the Steepest descent method.By looking at table 5 of example 3, one can see the last iteration for the Conjugate gradient method is  14 = [9.9914× 10 −9 , 9.3203 × 10 −8 ]  and the last iteration of the Steepest descent method is  32 = [4.6409× 10 −6 , −2.0057 × 10 −6 ]  .The results obtained from the Conjugate gradient method have more accuracy than the Steepest descent method, according to   values.Similarly, the results of Example 2, presented in table 3 and table 4, show that the Conjugate gradient method has more accuracy than the Steepest descent method.Hence, we can conclude that the Conjugate gradient method is more accurate than the Steepest descent method.
Algorithms for both methods have been presented.Matlab program has been implemented for solving difficult examples.Matlab codes have been presented for minimizing three examples by both methods.Moreover, a comparison has been madebetween the Conjugate gradient method and the Steepest descent method.A discussion has been carried out based on the results.The provided results demonstrates that, the Conjugate gradient method requires fewer iterations and is more efficient than the Steepest descent method.On the other hand, The Steepest descent approach takes less time to converge a function than the Conjugate gradient method.The authors declare no conflict of interest.