We explore the connections between linear algebra and calculus. This is necessary in order to generalize calculus concepts to functions of several variables; since any interesting management problem involves multiple dimensions, this is a worthy task. The simplest nonlinear function of multiple variables is arguably a quadratic form:
Denoting the double sum as 
is typically preferred to 
, as the latter is a bit ambiguous. To see the issue, consider a term such as 4x1x2; we could rewrite this as 3x1x2 + x2x1 or 2x1x2 + 2x2x1. The notation above results in a unique way of expressing a quadratic form. Concrete examples of quadratic forms are
Quadratic forms are important for several reasons:
- They play a role in the approximation of multivariable functions using Taylor expansions; in Section 3.9 we will see how we can generalize the Taylor expansion of Section 2.10.
- Quadratic forms are also important in nonlinear optimization (see Section 12.5).
- In probability and statistics, the variance of a linear combination of random variables is linked to a quadratic form (Section 8.3); this is fundamental, among many other things, in financial portfolio management.
Quadratic forms are strongly linked to linear algebra as they can be most conveniently associated with a symmetric matrix. In fact, we may also express any quadratic form as
where matrix A is symmetric. For instance, we can express the quadratic forms in Eq. (3.18) as follows:
Note that cross-product terms correspond to off-diagonal entries in the matrix; in fact, the aij coefficients in expression (3.17), for i ≠ j, occur divided by 2 in the matrix. These two quadratic forms are plotted in Figs. 3.10 and 3.11, respectively. From what we know about convexity (Section 2.11), we immediately see that f is convex, whereas g is neither convex nor concave, as it features a “saddle point.” Convexity and concavity of a quadratic form are linked to properties of the corresponding matrix.
Fig. 3.10 A convex quadratic form.
Fig. 3.11 An undefined quadratic form.
DEFINITION 3.11 (Definiteness of quadratic forms) We say that a quadratic form xTAx
- Is positive definite if xTAx > 0 for all x ≠ 0
- Is positive semidefinite if xTAx ≥ 0 for all x
- Is negative definite if xTAx < 0 for all x ≠ 0
- Is negative semidefinite if xTAx ≤ 0 for all x
Otherwise, we say that the quadratic form is indefinite.
The definiteness of a quadratic form is strictly related to convexity by the following properties:
- A positive definite quadratic form is a strictly convex function.
- A positive semidefinite quadratic form is a convex function.
- A negative definite quadratic form is a strictly concave function.
- A negative semidefinite quadratic form is a concave function.
Incidentally, we may observe that for a positive definite quadratic form, the origin x* = 0 is a global minimizer, as it is the only point at which the function is zero, whereas it is strictly positive for any x ≠ 0. By the same token, x* = 0 is a global maximizer for a negative definite quadratic form. For semidefinite forms, there may be alternative optima, i.e., several vectors x for which the function is zero.
Now the main problem is to find a way to check the definiteness of a quadratic form. A good starting point is to observe that this task is easy to accomplish for a diagonal matrix:
In such a case, the quadratic form does not include cross-products:
It is easy to see that definiteness of this quadratic form depends on the signs of the diagonal terms λi. If they are all strictly positive (negative), the form is positive (negative) definite. If they are nonnegative (nonpositive), the form is positive (negative) semidefinite. But for a diagonal matrix, the entries on the diagonal are just its eigenvalues. Since the matrix associated with a quadratic form is symmetric, we also know that all of its eigenvalues are real numbers. Can we generalize and guess that definiteness depends on the sign of the eigenvalues of the corresponding symmetric matrix? Indeed, this is easily verified by factorizing the quadratic form using Eq. (3.15)
where y = PTx. This is just another quadratic form, which has been diagonalized by a proper change of variables. We may immediately conclude that a quadratic form
- Is positive definite if and only if the eigenvalues of the corresponding matrix A are all strictly positive (> 0)
- Is positive semidefinite if and only if the eigenvalues of the corresponding matrix A are all nonnegative (≥ 0)
- Is negative definite if and only if the eigenvalues of the corresponding matrix A are all strictly negative (< 0)
- Is negative semidefinite, if and only if the eigenvalues of the corresponding matrix A are all nonpositive (≤ 0)
Leave a Reply