Differentiation of Real Scalar Fields

Directional Differentiability

Definition: Directional Derivative of a Real Scalar Field

Let $f:\mathcal{D}\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$, let ${\mathbf{x}}_0\in \mathcal{D}$ and let $\hat{\mathbf{r}}\in {\mathbb{R}}^n$ be some unit vector.

The directional derivative of $f$ at ${\mathbf{x}}_0$ along $\hat{\mathbf{r}}$ is the limit

$$\underset{h\to 0}{\lim }\frac{f({\mathbf{x}}_0+h\cdot \hat{\mathbf{r}})-f({\mathbf{x}}_0)}{h},$$

if it exists.

NOTATION

$$\frac{\partial f}{\partial \hat{\mathbf{r}}}({\mathbf{x}}_0){\partial }_{\hat{\mathbf{r}}}f({\mathbf{x}}_0)f_{\hat{\mathbf{r}}}({\mathbf{x}}_0)D_{\hat{\mathbf{r}}}f({\mathbf{x}}_0)$$

Partial Differentiability

Definition: Partial Derivatives of a Real Scalar Field

Let $f:\mathcal{D}\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$, let $\mathbf{a}\in \mathcal{D}$, let $\varphi :\mathcal{D}\to {\mathbb{R}}^n$ be a coordinate system on $\mathcal{D}$ and let $a^1={\varphi }^1(\mathbf{a}),\dots ,a^n={\varphi }^n(\mathbf{a})$ be the coordinates of $\mathbf{a}$ in $\varphi$.

The partial derivative of $f$ at $\mathbf{a}$ with respect to the $i$-th coordinate ${\varphi }^i$ is the limit

$$\underset{h\to 0}{\lim }\frac{f(a^1,\dots ,a^i+h,\dots ,a^n)-f(a^1,\dots ,a^i,\dots ,a^n)}{h},$$

NOTATION

Most commonly, the partial derivative of $f$ at $\mathbf{a}$ with respect to the $i$-th coordinate ${\varphi }^i$ is denoted as:

$$\frac{\partial }{\partial {\varphi }^i}(\mathbf{a})\frac{\partial f}{\partial {\varphi }^i}f(\mathbf{a}){\frac{\partial f}{\partial {\varphi }^i}|}_{\mathbf{a}}{f}_{{\varphi }^i}(\mathbf{a})$$

If the coordinate system is evident from context, we can also write ${\partial }_{i}f(\mathbf{a})$.

Note: Partial Derivative as a Function

When there is no specific $\mathbf{a}\in \mathcal{D}$ mentioned, the term “partial derivative” refers to the the function $\frac{\partial f}{\partial {\varphi }^i}:\mathcal{D}\to \mathbb{R}$ which to each $\mathbf{x}\in \mathcal{D}$ assigns the partial derivative of $f$ at $\mathbf{a}$ with respect to the coordinate ${\varphi }^i$.

Note: Orders of Partial Derivatives

An $n$-th order partial derivative of $f$ is a partial derivative of $f$‘s $(n-1)$-th partial derivative function.

Definition: Partial Differentiability of Real Scalar Fields

We say that $f$ is $k$-times (continuously) partially differentiable at $\mathbf{a}\in \mathcal{D}$ if all of its $k$-th order partial derivatives with respect to all coordinates of $\varphi$ exist at $\mathbf{a}$ (and are continuous there).

If the above is true for every $\mathbf{a}$ in some $S\subseteq \mathcal{D}$, then we say that $f$ is $k$-times (continuously) partially differentiable on $S$. We can also omit “on $S$” when $S=\mathcal{D}$.

NOTE

If there is no specific coordinate system mentioned, then we mean that $f$ is $k$-times (continuously) partially differentiable with respect to Cartesian coordinates.

Warning: Partial Differentiability's Dependence on Coordinate Systems

If $f$ is partially differentiable in one coordinate system, then that does not necessarily mean that it is partially differentiable in every coordinate system.

Example

Consider the function $f:{\mathbb{R}}^2\to \mathbb{R}$ defined in Cartesian coordinates as

$$f(x,y)\stackrel{\text{def}}{=}\{\begin{array}{l}{x}^2+y^2\text{ if }y\ne 0\\ 0\text{ if }y=0\end{array}$$

Let’s see what $f$‘s partial derivatives with respect to $x$ and $y$ are.

Partial derivative with respect to $x$:

For $y\ne 0$,

$$\frac{\partial }{\partial x}f(x,y)=\frac{\partial f}{\partial x}{x}^2+y^2=2x$$

For $y=0$, we have $f(x,0)=0$ and thus

$$\frac{\partial }{\partial x}f(x,y)=\underset{h\to 0}{\lim }\frac{f(x+h,0)-f(x,0)}{h}=\underset{h\to 0}{\lim }\frac{0-0}{h}=0$$

So, the partial derivative of $f$ with respect to $x$ exists at every point.

Partial derivative with respect to $y$:

For $y\ne 0$,

$$\frac{\partial }{\partial y}f(x,y)=\frac{\partial }{\partial y}(x^2+y^2)=2y.$$

For $y=0$, again we have $f(x,0)=0$ and so

$$\frac{\partial }{\partial y}f(x,y)=\underset{h\to 0}{\lim }\frac{f(x,0+h)-f(x,0)}{h}=\underset{h\to 0}{\lim }\frac{(x^2+h^2)-0}{h}=\underset{h\to 0}{\lim }\frac{x^2+h^2}{h}$$

However, this limit does not exists, since it depends on whether $h$ approaches $0$ from the left or the right.

Therefore, $f$ is not partially differentiable with respect to $y$ on the points $(x,0)$.

Now, let’s take a look at another coordinate system $u,v$. The transformations between $(u,v)$ and $(x,y)$ are given below.

$$\begin{array}{rl}& u=x+yv=x-y\\ & x=\frac{u+v}{2}y=\frac{u-v}{2}\end{array}$$

When $u=v$, i.e. when $y=0$, the function $f$ is also $0$.

When $u\ne v$, the function $f$ expressed in the coordinate system $(u,v)$ is

$$\begin{array}{rl}f(u,v)& ={\left(\frac{u+v}{2}\right)}^2+{\left(\frac{u-v}{2}\right)}^2\\ & =\frac{u^2+2uv+v^2}{4}+\frac{u^2-2uv+v^2}{4}\\ & =\frac{2u^2+2v^2}{4}\\ & =\frac{u^2+v^2}{2}\end{array}$$

Therefore,

$$f(u,v)=\{\begin{array}{l}\frac{u^2+v^2}{2}u\ne v\\ 0u=v\end{array}$$

Let’s examine the partial derivatives of $f$ with respect to $u$ and $v$.

Partial derivative with respect to $u$:

For $u\ne v$,

$$\frac{\partial }{\partial u}f(u,v)=\frac{\partial }{\partial u}\frac{u^2+v^2}{2}=\frac{1}{2}\times 2u=u$$

For $u=v$, we have $f(u,v)=0$

$$\frac{\partial }{\partial u}f(u,v)=\underset{h\to 0}{\lim }\frac{f(u+h,v)-f(u,v)}{h}=\underset{h\to 0}{\lim }\frac{\frac{(u+h{)}^2+v^2}{2}-0}{h}=\underset{h\to 0}{\lim }\frac{(u+h{)}^2+v^2}{2h}=0$$

Therefore, the partial derivative of $f$ with respect to $u$ exists at every point.

Partial derivative with respect to $v$:

For $u\ne v$,

$$\frac{\partial }{\partial v}f(u,v)=\frac{\partial }{\partial v}\frac{u^2+v^2}{2}=\frac{1}{2}\times 2v=v$$

For $u=v$, we have $f(u,v)=0$

$$\frac{\partial }{\partial v}f(u,v)=\underset{h\to 0}{\lim }\frac{f(u,v+h)-f(u,v)}{h}=\underset{h\to 0}{\lim }\frac{\frac{u^2+(v+h{)}^2}{2}-0}{h}=\underset{h\to 0}{\lim }\frac{u^2+(v+h{)}^2}{2h}=0$$

Therefore, the partial derivative of $f$ with respect to $v$ exists at every point.

This means that $f$ is partially differentiable at every point with respect to the coordinate system $(u,v)$, but it is not partially differentiable with respect to $(x,y)$.

Theorem: Partial Derivatives in Cartesian Coordinates

Let $f:\mathcal{D}\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$.

The partial derivative of a real scalar field $f$ at $\mathbf{a}$ with respect to the $i$-th Cartesian coordinate $x^i$ coincides with the directional derivative of $f$ along the $i$-th standard basis vector ${\mathbf{e}}_i$:

$$\frac{\partial f}{\partial x^i}(\mathbf{a})=\underset{h\to 0}{\lim }\frac{f(\mathbf{a}+h\cdot {\mathbf{e}}_i)-f(\mathbf{a})}{h}$$

PROOF

TODO

NOTATION

Cartesian coordinates in partial derivatives can be denoted using either subscripts and superscripts, i.e.

$$\frac{\partial }{\partial x^i}(\mathbf{a})\frac{\partial f}{\partial x^i}f(\mathbf{a}){\frac{\partial f}{\partial x^i}|}_{\mathbf{a}}{f}_{x^i}(\mathbf{a})$$ $$\frac{\partial }{\partial x_i}(\mathbf{a})\frac{\partial f}{\partial x_i}f(\mathbf{a}){\frac{\partial f}{\partial x_i}|}_{\mathbf{a}}{f}_{x_i}(\mathbf{a})$$

are equivalent. Writing the coordinates using subscripts is cleaner when partial derivatives of higher orders are involved.

Schwarz's Theorem: Symmetry of Second-Order Partial Derivatives

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}$, let $\mathbf{a}\in \mathcal{D}$.

If $f$ is twice continuously partially differentiable at $\mathbf{a}$, then for all $i,j\in \{1,\cdots ,n\}$

$$\frac{\partial }{\partial x^i}\frac{\partial }{\partial x^j}f(\mathbf{a})=\frac{\partial }{\partial x^j}\frac{\partial }{\partial x^i}f(\mathbf{a})$$

PROOF

TODO

Gradient

Definition: Gradient

Let $f:\mathcal{D}\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$ and let $\mathbf{a}\in \mathcal{D}$.

If $f$ is partially differentiable at $\mathbf{a}$ with respect to Cartesian coordinates, then the gradient of $f$ at $\mathbf{a}$ is the following vector:

$$\left[\begin{array}{c}\frac{\partial f}{\partial x^1}(\mathbf{a})\\ ⋮\\ \frac{\partial f}{\partial x^n}(\mathbf{a})\end{array}\right]$$

NOTATION

$$\nabla f(\mathbf{a})\mathrm{grad}f(\mathbf{a})$$

The gradient at $\mathbf{a}$ shows the directions in which small deviations from $\mathbf{a}$ result in the largest increase and the largest decrease in the value of $f$:

• An infinitesimally small deviation from $\mathbf{a}$ in the direction of $\nabla f(\mathbf{a})$ will result in the greatest possible increase in the value of $f$. If the deviation from $\mathbf{a}$ is in any other direction, then the increase in the value of $f$ will necessarily be smaller (closer to 0).
• An infinitesimally small deviation from $\mathbf{a}$ in the direction of $-\nabla f(\mathbf{a})$ will result in the greatest possible decrease in the value of $f$. If the deviation from $\mathbf{a}$ is in any other direction, then the decrease in the value of $f$ will necessarily be smaller (closer to 0).

Hessian Matrix

Definition: Hessian Matrix

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field which is twice partially differentiable with respect to Cartesian coordinates.

The Hessian matrix of $f$ is the $n\times n$-matrix $H_f$ whose columns are the gradients of $f$‘s partial derivatives:

$$H_f\stackrel{\text{def}}{=}\left[\begin{array}{ccc}|& |& |\\ \nabla ({\partial }_{1}f)& \cdots & \nabla ({\partial }_{n}f)\\ |& |& |\end{array}\right]=\left[\begin{array}{ccc}{\partial }_1{\partial }_{1}f& \cdots & {\partial }_1{\partial }_{n}f\\ ⋮& \ddots & ⋮\\ {\partial }_n{\partial }_{1}f& \cdots & {\partial }_n{\partial }_{n}f\end{array}\right]$$

NOTATION

The Hessian matrix $H_f$ is different for different $\mathbf{x}\in \mathcal{D}$, since the partial derivatives of $f$ depend on $\mathbf{x}$. The Hessian matrix at a particular $\mathbf{a}\in \mathcal{D}$ is thus denoted as $H_f(\mathbf{a})$ to make this dependency apparent.

Theorem: Symmetry of the Hessian Matrix

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a differentiable real scalar field with differentiable partial derivatives.

If all of second partial derivatives of $f$‘s second-order partial derivatives are also continuous, then the Hessian matrix of $f$ is symmetric for every $\mathbf{x}\in D$.

PROOF

TODO

Differentiability

We can use partial derivatives to provide an alternative definition for the differentiability of real scalar fields.

Definition: Differentiability of Real Scalar Fields

Let $f:\mathcal{D}\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$ and let $\mathbf{a}\in \mathcal{D}$.

We say that $f$ is differentiable at $\mathbf{a}$ if and only if $f$ is partially differentiable at $\mathbf{a}$ with respect to Cartesian coordinates and the following limit is zero:

$$\underset{\mathbf{x}\to \mathbf{a}}{\lim }\frac{f(\mathbf{x})-[f(\mathbf{a})+\nabla f(\mathbf{a})\cdot (\mathbf{x}-\mathbf{a})]}{||\mathbf{x}-\mathbf{a}||}=0,$$

where $\nabla f(\mathbf{a})\cdot (\mathbf{x}-\mathbf{a})$ is the dot product between $f$‘s gradient at $\mathbf{a}$ and $(\mathbf{x}-\mathbf{a})$.

Tip: Gradient and Jacobian

The gradient $\nabla f(\mathbf{a})$ is then just the transpose of $f$‘s Jacobian matrix.

Definition: Critical Point

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field.

We say that $f$ has a critical point at $\mathbf{a}\in \mathcal{D}$ if $f$ is not differentiable at $\mathbf{a}$ or it is differentiable at $\mathbf{a}$ but its gradient there is zero.

$$\nabla f(\mathbf{a})=\vec{0}$$

Theorem: Differentiability implies Directional Differentiability

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$ and let $\mathbf{a}\in \mathcal{D}$.

If $f$ is differentiable at $\mathbf{a}$, then its directional derivatives ${\partial }_{\hat{\mathbf{r}}}f(\mathbf{a})$ at $\mathbf{a}$ exist along every direction $\hat{\mathbf{r}}$ and are equal to the dot product of $f$‘s gradient with $\hat{\mathbf{r}}$:

$$(\nabla f(\mathbf{a}))\cdot \hat{\mathbf{r}}={\partial }_{\hat{\mathbf{r}}}f(\mathbf{a})$$

PROOF

TODO

Theorem: Chain Rule for Scalar Fields

Let $f:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be a real scalar field on an open subset $\mathcal{D}\subseteq {\mathbb{R}}^n$ and let $\mathbf{a}\in \mathcal{D}$. Let $\mathbf{r}:I\to \mathcal{D}$ be a vector-valued function on an open subset $I\subseteq \mathbb{R}$ and let $t\in I$.

If $\mathbf{r}$ is differentiable at $t$ and $f$ is differentiable at $\mathbf{r}(t)$, then the derivative of the composition $f\circ \mathbf{r}:I\to \mathbb{R}$ is the dot product of $f$‘s gradient and $\mathbf{r}$‘s derivative .

$$\frac{\mathrm{d}}{\mathrm{d}t}f(\mathbf{r}(t))=\nabla f(\mathbf{r}(t))\cdot {\mathbf{r}}^{\prime }(t)$$

PROOF

TODO

Theorem: Product Rule

Let $f:{\mathcal{D}}_f\subseteq {\mathbb{R}}^n\to \mathbb{R}$ and $f:{\mathcal{D}}_g\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be real scalar fields and let $\mathbf{a}\in {\mathcal{D}}_f\cap {\mathcal{D}}_g$.

If $f$ and $g$ are differentiable at $\mathbf{a}$, then the product $fg$ is also differentiable at $\mathbf{a}$ and its gradient is the following:

$$\nabla (fg)(\mathbf{a})=g(\mathbf{a})\nabla f(\mathbf{a})+f(\mathbf{a})\nabla g(\mathbf{a})$$

PROOF

TODO

Theorem: Quotient Rule

Let $f:{\mathcal{D}}_f\subseteq {\mathbb{R}}^n\to \mathbb{R}$ and $f:{\mathcal{D}}_g\subseteq {\mathbb{R}}^n\to \mathbb{R}$ be real scalar fields and let $\mathbf{a}\in {\mathcal{D}}_f\cap {\mathcal{D}}_g$.

If $f$ and $g$ are differentiable at $\mathbf{a}$ and $g(\mathbf{a})\ne 0$, then the quotient $f/g:\mathcal{D}\subseteq {\mathbb{R}}^n\to \mathbb{R}$ is also differentiable at $\mathbf{a}\in \mathcal{D}$ its gradient is the following:

$$\nabla (f/g)(\mathbf{a})=\frac{g(\mathbf{a})\nabla f(\mathbf{a})-f(\mathbf{a})\nabla g(\mathbf{a})}{g(\mathbf{a}{)}^2}$$

PROOF

TODO