Lagrange Multipliers

You want to find the largest or smallest value of a function, but you are not free to move in every direction. A constraint restricts you to a curve, a surface, or some other subset of the domain. Without the constraint, you would set the gradient to zero and look for critical points. With the constraint in place, the optimum rarely falls at a critical point of the function itself. It lies on the constraint set, and the condition for a critical point no longer identifies it. The method of Lagrange multipliers gives a condition that does.

Think in two dimensions. You want to maximize $f(x, y)$ subject to $g(x, y) = 0$. The constraint traces out a curve in the plane. As you move along this curve, $f$ increases and decreases. At the point where $f$ reaches its constrained maximum, moving along the curve in either direction cannot increase $f$. That means $\nabla f$ has no component tangent to the curve at that point. It points entirely in the normal direction. But $\nabla g$ also points normal to the curve, because the gradient of any function is perpendicular to its level sets. So at the constrained optimum, $\nabla f$ and $\nabla g$ must be parallel. Parallel means one is a scalar multiple of the other, so $\nabla f = \lambda \nabla g$ for some scalar $\lambda$. That scalar is the Lagrange multiplier. Together with $g(x, y) = 0$, this makes three equations in three unknowns, $x$, $y$, and $\lambda$.

Take a concrete case. Maximize $f(x, y) = x + y$ on the unit circle $x^2 + y^2 = 1$. Set $g(x, y) = x^2 + y^2 - 1$. The gradients are $\nabla f = (1,\, 1)$ and $\nabla g = (2x,\, 2y)$. The condition $\nabla f = \lambda \nabla g$ gives

\[1 = 2\lambda x, \quad 1 = 2\lambda y.\]

These two equations force $x = y$. Substituting into $x^2 + y^2 = 1$ gives $2x^2 = 1$, so $x = y = 1/\sqrt{2}$. The multiplier is $\lambda = 1/(2x) = \sqrt{2}/2$. The maximum value of $x + y$ on the unit circle is $1/\sqrt{2} + 1/\sqrt{2} = \sqrt{2}$. The other solution, $x = y = -1/\sqrt{2}$, gives the minimum, $-\sqrt{2}$.

The multiplier $\lambda$ is not just a device for solving the system. It measures how sensitive the optimal value is to the constraint. If you perturbed the constraint to $x^2 + y^2 = 1 + \epsilon$ for a small $\epsilon$, the maximum of $f$ would shift by approximately $\lambda \epsilon$. A large multiplier means the optimal value is highly sensitive to the constraint. A multiplier of zero means the optimal value does not change to first order, and the constrained optimum is already a critical point of $f$.