Module 4 : Solving Linear Algebraic Equations
Section 11 : Appendix C: Steepest Descent / Gradient Search Method
 
The first step in the line search is to find bounds on the optimal step size $\lambda ^{\ast }.$ These are established by finding two points, say $\alpha $ and $\beta $, such that the slope $d\phi /d\lambda $
MATH ------(254)
------(255)
has opposite signs at these points. We know that at $\lambda =0,$
MATH ------(256)

as $s^{(k)}$ is assumed to be a descent direction. Thus, we take $\alpha $ corresponding to $\lambda =0$ and try to find a point $\lambda =$ $\beta $ such that MATH The point $\beta $ can be taken as the first value out of $\lambda =h,$ $2h,4h,8h,....$ for which MATHwhere $h$ is some pre-assigned initial step size. As $d\phi /d\lambda $ changes sign in the interval $[0,\beta ],$ the optimum $\lambda ^{\ast }$ is bounded in the interval $[0,\beta ].$

The next step is to approximate $\phi (\lambda )$ over interval $[0,\beta ]$ by a cubic interpolating polynomial for the form

MATH ------(257)

The parameters $a$ and $b$ be computed as MATHThe parameters $c$ and $d$ can be computed by solving

MATHi.e. by solving MATHThe application of necessary condition for optimality yields

MATH ------(258)
i.e.
MATH ------(259)
One of the two values correspond to the minimum. The sufficiency condition for minimum requires
MATH ------(260)

The fact that $d\phi /d\lambda $ has opposite sign at $\lambda =0$ and $\lambda =\beta $ ensures that the equation 258 does not have imaginary roots.

Table 8: Line Search using Cubic Interpolation Algorithm
INITIALIZE: MATH
Step 1: Find $\beta $
$\qquad \beta =h$
WHILE MATH
MATH
END WHILE
Step 2: Solve for $a,b,c$ and $d$ using MATH and $\beta $
Step 3: Find $\lambda ^{\ast }$ using sufficient condition for optimality