Extrema of Real Scalar Fields

Local Extrema

Definition: Local Minimum

Let $f:D\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field.

We say that $f({\vec{x}}_0)$ is a local minimum of $f$ if there is an open ball $B_{\epsilon }({\vec{x}}_0)$ around ${\vec{x}}_0\in D$ where $f({\vec{x}}_0)$ is the smallest funtional value.

$f({\vec{x}}_0)\le f(\vec{x})\forall \vec{x}\in B_{\epsilon }({\vec{x}}_0)$

We say that ${\vec{x}}_0$ is a place of a local minimum for $f$.

Definition: Global Maximum

Let $f:D\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field.

We say that $f({\vec{x}}_0)$ is a local maximum of $f$ if there is an open ball $B_{\epsilon }({\vec{x}}_0)$ around ${\vec{x}}_0\in D$ where $f({\vec{x}}_0)$ is the greatest funtional value.

$$f({\vec{x}}_0)\ge f(\vec{x})\forall \vec{x}\in B_{\epsilon }({\vec{x}}_0)$$

We say that ${\vec{x}}_0$ is a place of a local maximum for $f$.

Definition: Local Extremum

The local minima and local maxima] of a real scalar field are collectively known as its local extrema.

Global Extrema

Definition: Global Minimum

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field and let ${\mathbf{x}}_0\in \mathcal{D}$.

We call ${\mathbf{x}}_0$ a place of a global minimum of $f$ iff

$$f({\mathbf{x}}_0)\le f(\mathbf{x})\forall \mathbf{x}\in \mathcal{D}$$

The value $f({\mathbf{x}}_0)$ is known as the global minimum of $f$.

Theorem: Uniqueness of the Global Minimum

If $f$ has a global minimum, then its value is unique.

PROOF

TODO

NOTE

The global minimum is unique but may occur at multiple places.

Definition: Global Maximum

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field and let ${\mathbf{x}}_0\in \mathcal{D}$.

We say that $f({\mathbf{x}}_0)$ is the global maximum of $f$ if it is the greatest functional value:

We call ${\mathbf{x}}_0$ a place of a global maximum of $f$ iff.

$$f({\mathbf{x}}_0)\ge f(\mathbf{x})\forall \mathbf{x}\in \mathcal{D}$$

The value $f({\mathbf{x}}_0)$ is known as the global maximum of $f$.

Theorem: Uniqueness of the Global Maximum

If $f$ has a global maximum], then its value is unique.

PROOF

TODO

NOTE

The global maximum is unique but may occur at multiple places.

Definition: Global Extremum

The global minimum and the global maximum of a real scalar field are collectively known as its global extrema.

Saddle Points

Definition: Saddle

Let $f:D\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field.

We say that $f$ has a saddle point at ${\vec{x}}_0\in D$ if in every open ball $B_r({\vec{x}}_0)$ around ${\vec{x}}_0$ there are $\vec{a}$ and $\vec{b}$ such that

$$f(\vec{a})<f({\vec{x}}_0)<f(\vec{b})$$

Finding Extrema

Theorem: Finding Local Extrema

Let $f:\mathcal{D}\to \mathbb{R}$ be a real scalar field.

If $f$ has a local extremum at $\mathbf{a}\in \mathcal{D}$, then $\mathbf{a}$ is a critical point of $f$.

PROOF

TODO

Theorem: Hessian Matrix Criteria for Local Extrema

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field which is twice continuously partially differentiable in Cartesian coordinates on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$.

A critical point $\mathbf{p}\in \mathcal{D}$ is:

• a place of a local maximum if the Hessian matrix $H_f(\mathbf{p})$ is negative-definite;
• a place of a local minimum if the Hessian matrix $H_f(\mathbf{p})$ is positive-definite;
• a place of a saddle point if the Hessian matrix $H_f(\mathbf{p})$ is indefinite.

If the Hessian matrix $H_f(\mathbf{p})$ is semi-definite, then it cannot be used to make any predictions.

PROOF

TODO

Constraints

In practice, it is rare that we simply need to find the extrema of a real scalar field. Usually, we are interested in finding extrema under certain conditions.

EXAMPLE

Imagine you are tasked with the construction of a box with a lid which should be able to fit exactly $1$ cubic metre of stuff. You want to minimise the cost and thus the amount of paperboard used for the box. In other words, you want to find what dimensions of the box which require the least material. The amount of material is given by

$$p(x,y,z)=\underset{\text{front and back}}{2xy}+\underset{\text{sides}}{2yz}+\underset{\text{bottom and lid}}{2xz}$$

You can check that, by itself, this function has neither local nor global extrema. However, we also have another condition - the box should have a volume of $1$ cubic metre. This gives us a constraint for the dimensions $x,y,z$:

$$xyz=1$$

This allows us to the express one variable using the other three and obtain an expression for $p$ which has only 2 variables.

$$z=\frac{1}{xy}$$ $$p(x,y)=2xy+2y\times \frac{1}{xy}+2x\times \frac{1}{xy}=2xy+\frac{2}{x}+\frac{2}{y}$$

This function does have a local minimum, which occurs at $x=1$ and $y=1$. Using $z=\frac{1}{xy}$, we find that $z=1$. Thus, a cube is the box shape which requires the least amount of material to fit $1$ cubic metre of stuff.

Definition: Equality Constraint

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field.

An equality constraint is an equation of the form

$$g(\mathbf{x})=c$$

for some real scalar field $g:\mathcal{D}\to \mathbb{R}$ and some constant $c\in \mathbb{R}$.

INTUITION

Constraints restrict the part of the domain of $f$ in which we are interested. Although $f$ itself might not admit extrema on its entire domain, when restricted to only those values for which the constraint is fulfilled, this might change might. In other words, $f$ might not have extrema, but its restriction $f{|}_S$ on $S=\{\mathbf{x}\mid g(\mathbf{x})=c\}$ might.

Theorem: Lagrange Multipliers

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a continuously partially differentiable real scalar field and let

$$\begin{array}{rl}{g}_1(\mathbf{x})& =c_1\\ & ⋮\\ g_k(\mathbf{x})& =c_k\end{array}$$

be some constraints, where $g_1,\dots ,g_k:\mathcal{D}\to \mathbb{R}$ are also continuously partially differentiable, and let $S\{\mathbf{x}\mid g_1(\mathbf{x})=c_1\text{ and }\cdots \text{ and }{g}_k(\mathbf{x})=c_k\}$.

If $f{|}_S$ has a local extremum at $\mathbf{a}$ and the gradients $\nabla g_1(\mathbf{a}),\dots ,\nabla g_k(\mathbf{a})$ are linearly independent, then $\nabla f(\mathbf{a})$ is a linear combination of $\nabla g_1(\mathbf{a}),\dots ,\nabla g_k(\mathbf{a})$.

PROOF

TODO

Definition: Lagrange Multipliers

The coefficients ${\lambda }_1,\dots ,{\lambda }_k$ in the representation

$$\nabla f(\mathbf{a})={\lambda }_1\nabla g_1(\mathbf{a})+\cdots +{\lambda }_k\nabla g_k(\mathbf{a})$$

of $\nabla f(\mathbf{a})$ as a linear combination of $\nabla g_1(\mathbf{a}),\dots ,\nabla g_k(\mathbf{a})$ are known as Lagrange multipliers.

EXAMPLE

TODO