Laplace Approximation

Not every integral has a closed form. Some integrands rise to a sharp peak and drop off quickly on both sides, so nearly all the area lies in a narrow region around the peak. If you can describe the height and width of the peak, you can estimate the whole integral without evaluating it exactly. The Laplace approximation does this by fitting a Gaussian to the peak. A Gaussian integral has a known closed form, so the estimate comes out as an explicit formula.

Suppose you need to evaluate $\int e^{-f(x)}\, dx$ where $f$ is smooth and has a unique minimum at $x_0$. At the minimum, the first derivative vanishes, so expanding $f$ to second order gives $f(x) \approx f(x_0) + \frac{1}{2}\, f^{\prime\prime}(x_0)\,(x - x_0)^2$. Substituting this into the integrand turns it into a Gaussian centered at $x_0$. The integral of that Gaussian over the real line is known, and the result is

\[\int e^{-f(x)}\, dx \;\approx\; e^{-f(x_0)}\,\sqrt{\frac{2\pi}{f''(x_0)}}\,.\]

The factor $e^{-f(x_0)}$ captures the height of the peak. The square root captures its width. A large second derivative means a narrow peak and a smaller integral. A small second derivative means a wide peak and a larger one. The behavior of the integrand away from the peak is discarded, replaced by the assumption that the Gaussian fit is close enough over the region that contributes most to the integral. For a function of several variables with a minimum at $\mathbf{x}_0$, the single second derivative is replaced by the Hessian matrix $H$, and the formula becomes $e^{-f(\mathbf{x}_0)}\,\sqrt{(2\pi)^n / \det H}$.

Take a concrete case. Approximate $I = \int_0^{\infty} x^{10}\, e^{-x}\, dx$. The integrand peaks near $x = 10$ and falls off steeply on both sides. Write $x^{10}\, e^{-x}$ as $e^{10 \ln x - x}$ to put it in the form $e^{-f(x)}$ with $f(x) = x - 10 \ln x$. The derivative $f^{\prime}(x) = 1 - 10/x$ vanishes at $x_0 = 10$. The minimum value is $f(10) = 10 - 10\ln 10$. The second derivative $f^{\prime\prime}(x) = 10/x^2$ gives $f^{\prime\prime}(10) = 1/10$. The formula yields

\[I \;\approx\; e^{-(10 - 10\ln 10)}\,\sqrt{\frac{2\pi}{1/10}} \;=\; 10^{10}\, e^{-10}\,\sqrt{20\pi}\,.\]

This evaluates to roughly 3598696. The integral equals $10!$ by definition of the gamma function, and $10! = 3628800$. The approximation is within one percent of the exact value.

Replacing 10 with a general positive integer $n$ gives $n! \approx n^n\, e^{-n}\, \sqrt{2\pi n}$. This is Stirling’s formula. The same technique appears throughout Bayesian inference. A posterior distribution is often proportional to $e^{-f(\boldsymbol{\theta})}$ for some function of the parameters. The minimum of $f$ occurs at the most probable parameter values, and the Hessian there measures how tightly the posterior concentrates around them. The Laplace approximation replaces the full posterior with a Gaussian centered at this mode, reducing an intractable distribution to a mean vector and a covariance matrix. The fit improves as more data accumulates and the peak grows sharper.