Hessian Matrix

A function of one variable has a second derivative that measures how the slope changes. Positive means the graph bends upward. Negative means it bends downward. For a function of several variables, the slope can change at different rates in different directions, and a single number cannot capture this. The Hessian matrix is what replaces it. It holds every second-order partial derivative and describes the curvature of the function in every direction at once.

Suppose $f$ is a scalar function of $n$ variables with continuous second partial derivatives. The Hessian matrix $H$ is $n \times n$. The entry in row $i$ and column $j$ is $\partial^2 f / \partial x_i \, \partial x_j$.

\[H = \begin{bmatrix} \dfrac{\partial^2 f}{\partial x_1^2} & \cdots & \dfrac{\partial^2 f}{\partial x_1 \, \partial x_n} \\[6pt] \vdots & \ddots & \vdots \\[6pt] \dfrac{\partial^2 f}{\partial x_n \, \partial x_1} & \cdots & \dfrac{\partial^2 f}{\partial x_n^2} \end{bmatrix}\]

Because the order of differentiation does not matter when the derivatives are continuous, $\partial^2 f / \partial x_i \, \partial x_j$ equals $\partial^2 f / \partial x_j \, \partial x_i$ and the Hessian is symmetric.

The Hessian appears naturally in the second-order Taylor expansion. Near a point $\mathbf{a}$, the function behaves like

\[f(\mathbf{a} + \mathbf{h}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a})^\top \mathbf{h} + \tfrac{1}{2}\, \mathbf{h}^\top H(\mathbf{a})\, \mathbf{h}.\]

At a critical point, the gradient vanishes, the linear term drops out, and the quadratic term $\frac{1}{2}\,\mathbf{h}^\top H\, \mathbf{h}$ alone controls the local shape. If every eigenvalue of $H$ is positive, this quadratic form is positive in every direction, and the critical point is a local minimum. If every eigenvalue is negative, it is a local maximum. If some eigenvalues are positive and others negative, the surface curves up in some directions and down in others, and the critical point is a saddle.

Take a concrete case. Let $f(x, y) = x^3 - 3x + y^2$. The gradient is $(3x^2 - 3,\, 2y)$. Setting it to zero gives $y = 0$ and $x^2 = 1$, so there are two critical points, $(1, 0)$ and $(-1, 0)$. The second partial derivatives are $\partial^2 f / \partial x^2 = 6x$, $\partial^2 f / \partial y^2 = 2$, and $\partial^2 f / \partial x\,\partial y = 0$. The Hessian is

\[H = \begin{bmatrix} 6x & 0 \\ 0 & 2 \end{bmatrix}.\]

At $(1, 0)$, this becomes \(\bigl[\begin{smallmatrix} 6 & 0 \\ 0 & 2 \end{smallmatrix}\bigr]\) with eigenvalues 6 and 2. Both are positive, so the surface opens upward in every direction and $(1, 0)$ is a local minimum. At $(-1, 0)$, the Hessian becomes \(\bigl[\begin{smallmatrix} -6 & 0 \\ 0 & 2 \end{smallmatrix}\bigr]\) with eigenvalues $-6$ and 2. The surface curves downward along the $x$-direction and upward along the $y$-direction, making $(-1, 0)$ a saddle point.

For a function of one variable, the second derivative test checks the sign of a single number. The Hessian is the multivariable version of this test. There is now one curvature for each independent direction, and the eigenvalues of the Hessian are those curvatures. The eigenvectors point along the directions of greatest and least curvature. Knowing the eigenvalues at a critical point tells you not just whether the point is a minimum, a maximum, or a saddle, but how sharply the surface curves away from it along each principal axis.