Definition: Directional Differentiability of Real Scalar Fields
Let \(f: \mathcal{D} \subseteq \mathbb{R}^n \to \mathbb{R}\) be a real scalar field on an open subset \(\mathcal{D} \subseteq \mathbb{R}^n\), let \(\boldsymbol{p} \in \mathcal{D}\) and let \(\boldsymbol{r} \in \mathbb{R}^n\).
We say that \(f\) is differentiable at \(\boldsymbol{p}\) along \(\boldsymbol{r}\) if the limit
\[\lim_{h\to 0}\frac{f(\boldsymbol{p} + h \cdot \boldsymbol{r} ) - f(\boldsymbol{p})}{h}\]
exists. The value of this limit is known as \(f\)'s directional derivative at \(\boldsymbol{p}\) along \(\boldsymbol{r}\).
Notation
\[\frac{\partial f}{\partial \boldsymbol{r}}(\boldsymbol{p}) \qquad \partial_{\boldsymbol{r}}f(\boldsymbol{p}) \qquad f_{\boldsymbol{r}}(\boldsymbol{p}) \qquad D_{\boldsymbol{r}} f(\boldsymbol{p})\]
Theorem: Total Differentiability \(\implies\) Directional Differentiability
Let \(f: \mathcal{D} \subseteq \mathbb{R}^n \to \mathbb{R}\) be a real scalar field and let \(\boldsymbol{p} \in \mathcal{D}\).
If \(f\) is totally differentiable at \(\boldsymbol{p}\), then it is directionally differentiable at \(\boldsymbol{p}\) along all unit vectors \(\boldsymbol{r} \in \mathbb{R}^n\) and the directional derivative of \(f\) along \(\boldsymbol{r}\) is given by the dot product of \(f\)'s gradient and \(\boldsymbol{r}\):
\[\partial_{\boldsymbol{r}}f(\boldsymbol{p}) = \nabla f(\boldsymbol{p}) \cdot \boldsymbol{r}\]
Example: \(f(\boldsymbol{x}) = \boldsymbol{a}^{\mathsf{T}} \boldsymbol{x}\)
Consider the real scalar field \(f: \mathbb{R}^n \to \mathbb{R}\) defined as
\[f(\boldsymbol{x}) = \boldsymbol{a}^{\mathsf{T}}\boldsymbol{x}\]
for some fixed real vector \(\boldsymbol{a} = \begin{bmatrix} a^1 & \cdots & a^n \end{bmatrix}^{\mathsf{T}}\in \mathbb{R}^n\).
It is totally differentiable on \(\mathbb{R}^n\) and its gradient is the following:
\[\nabla f(\boldsymbol{x}) = \boldsymbol{a}\]
Its directional derivatives are thus the following:
\[\partial_{\boldsymbol{r}} f(\boldsymbol{x}) = \nabla f(\boldsymbol{x}) \cdot \boldsymbol{r} = \boldsymbol{a} \cdot \boldsymbol{r}\]
Example: \(f(\boldsymbol{x}) = \boldsymbol{x}^{\mathsf{T}} \boldsymbol{A} \boldsymbol{x}\)
Consider the real scalar field \(f: \mathbb{R}^n \to \mathbb{R}\) defined as
\[f(\boldsymbol{x}) = \boldsymbol{x}^{\mathsf{T}} \boldsymbol{A} \boldsymbol{x}\]
for some real matrix \(\boldsymbol{A} \in \mathbb{R}^{n \times n}\).
It is totally differentiable on \(\mathbb{R}^n\) and its gradient is the following:
\[\nabla f(\boldsymbol{x}) = (\boldsymbol{A} + \boldsymbol{A}^{\mathsf{T}})\boldsymbol{x}\]
Therefore, its directional derivatives are as follows:
\[\partial_{\boldsymbol{r}} f(\boldsymbol{x}) = \nabla f(\boldsymbol{x}) \cdot \boldsymbol{r} = \nabla f(\boldsymbol{x})^{\mathsf{T}} \cdot \boldsymbol{r} = \boldsymbol{x}^{\mathsf{T}}(\boldsymbol{A}^{\mathsf{T}} + \boldsymbol{A})\boldsymbol{r}\]
Example:
Consider the real scalar field \(f: \mathbb{R}^2 \to \mathbb{R}\) defined as follows:
\[f\left(x, y\right) = \begin{cases} \frac{x y^2}{x^2 + y^2} & \text{if } x \ne 0 \text{ and } y \ne 0 \\ 0 & \text{otherwise} \end{cases}\]
For its directional derivatives at \(\boldsymbol{0}\), we have:
\[\begin{aligned}\partial_{\boldsymbol{r}}f(\boldsymbol{0}) & = \lim_{h \to 0} \frac{f(\boldsymbol{0} + h\boldsymbol{r}) - f(\boldsymbol{0})}{h} \\ & = \lim_{h \to 0} \frac{\frac{(hr_1)(hr_2)^2}{h^2 r_1^2 + h^2 r_2^2} - 0}{h} \\ & = \lim_{h \to 0} \frac{\frac{h^3 r_1 r_2^2}{h^2(r_1^2 + r_2^2)}}{h} \\ & = \lim_{h \to 0} \frac{\frac{h r_1 r_2^2}{r_1^2 + r_2^2}}{h} \\ & = \lim_{h \to 0}\frac{hr_1 r_2^2}{h(r_1^2 + r_2^2)} \\ & = \lim_{h \to 0} \frac{r_1 r_2^2}{r_1^2 + r_2^2} \\ & = \frac{r_1 r_2^2}{r_1^2 + r_2^2} \\ & = f(\boldsymbol{r})\end{aligned}\]
Therefore, all of \(f\)'s directional derivatives \(\boldsymbol{0}\) exist. More specifically, its partial derivatives are thr following:
\[\frac{\partial f}{\partial x}(\boldsymbol{0}) = \partial_{\boldsymbol{e}_1} f(\boldsymbol{0}) = f(\boldsymbol{e}_1) = \frac{1 \cdot 0^2}{1^2 + 0^2} = 0\]
\[\frac{\partial f}{\partial y}(\boldsymbol{0}) = \partial_{\boldsymbol{e}_2} f(\boldsymbol{0}) = f(\boldsymbol{e}_2) = \frac{0 \cdot 1^2}{0^2 + 1^2} = 0\]
If \(f\) were totally differentiable at \(\boldsymbol{0}\), then its gradient would have thus been given by the following:
\[\nabla f(\boldsymbol{0}) = \begin{bmatrix}\partial_{x}f(\boldsymbol{0}) \\ \partial_y f(\boldsymbol{0})\end{bmatrix} = \begin{bmatrix}0 \\ 0\end{bmatrix} = \boldsymbol{0}\]
The directional derivatives of \(f\) would then have to be given by
\[\partial_{\boldsymbol{r}}f(\boldsymbol{0}) = \nabla f(\boldsymbol{0}) \cdot \boldsymbol{r} = \boldsymbol{0} \cdot \boldsymbol{r} = 0\]
but we just proved that \(f\)'s directional derivatives are given by the following:
\[\partial_{\boldsymbol{r}}f(\boldsymbol{0}) = f(\boldsymbol{r})\]
In general, this is not equal to \(0\) and so we have reached a contradiction. Therefore, \(f\) is not totally differentiable at \(\boldsymbol{0}\) even though all of its directional derivatives there exist.
To confirm this, we assume that \(f\) is totally differentiable at \(\boldsymbol{0}\). Therefore, there its gradient \(\nabla f(\boldsymbol{0})\) exists and, since we just showed that \(\nabla f(\boldsymbol{0}) = \boldsymbol{0}\), we know that \(f(\boldsymbol{0} + \boldsymbol{h}) - f(\boldsymbol{0}) - \nabla f(\boldsymbol{0})^{\mathsf{T}}\boldsymbol{h}\) is little o of \(||\boldsymbol{h}||\) for \(\boldsymbol{h} \to \boldsymbol{0}\):
\[f(\boldsymbol{0} + \boldsymbol{h}) - f(\boldsymbol{0}) - \nabla f(\boldsymbol{0})^{\mathsf{T}}\boldsymbol{h} = o(||\boldsymbol{h}||) \qquad \text{for} \qquad \boldsymbol{h} \to \boldsymbol{0}\]
\[f(\boldsymbol{h}) - 0 - \boldsymbol{0}^{\mathsf{T}}\boldsymbol{h} = o(||\boldsymbol{h}||) \qquad \text{for} \qquad \boldsymbol{h} \to \boldsymbol{0}\]
\[f(\boldsymbol{h}) = o(||\boldsymbol{h}||) \qquad \text{for} \qquad \boldsymbol{h} \to \boldsymbol{0}\]
This implies the following limit:
\[\lim_{\boldsymbol{h} \to \boldsymbol{0}} \frac{f(\boldsymbol{h})}{||\boldsymbol{h}||} = 0\]
Let's examine the following real vector sequence \((\boldsymbol{h}_k)_{k \in \mathbb{N}}\):
\[\boldsymbol{h}_k = \begin{bmatrix}\frac{1}{k} \\ \frac{1}{k}\end{bmatrix}\]
Its limit for \(k \to \infty\) is obviously \(\boldsymbol{0}\). Furthermore, we have the following [limit](../../Real%20Functions/Limits%20(Real%20Functions.md):
\[\begin{aligned}\lim_{k \to \infty} \frac{f(\boldsymbol{h}_k)}{||\boldsymbol{h}_k||} & = \lim_{k \to \infty} \frac{\frac{\frac{1}{k} \cdot \frac{1}{k^2}}{\frac{1}{k^2} + \frac{1}{k^2}}}{\sqrt{\frac{1}{k^2} + \frac{1}{k^2}}} \\ & = \lim_{k \to \infty} \frac{\frac{1/k^3}{2/k^2}}{\sqrt{2/k^2}} \\ & = \lim_{k \to \infty} \frac{\frac{1}{2k}}{\frac{\sqrt{2}}{k}} \\ & = \lim_{k \to \infty} \frac{1}{2\sqrt{2}} \\ & = \frac{\sqrt{2}}{4}\end{aligned}\]
Since \(\frac{\sqrt{2}}{4} \ne 0\), we know that \(\lim_{\boldsymbol{h} \to \boldsymbol{0}} \frac{f(\boldsymbol{h})}{||\boldsymbol{h}||} \ne 0\) and so we reached a contradiction. Therefore, \(f\) cannot be totally differentiable at \(\boldsymbol{0}\).
Proof
Since \(f\) is totally differentiable at \(\boldsymbol{p}\), we have that \(f(\boldsymbol{p} + \boldsymbol{h}) - f(\boldsymbol{p}) - \nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{h}\) is little o of \(||\boldsymbol{h}||\) for \(\boldsymbol{h} \to \boldsymbol{0}\):
\[f(\boldsymbol{p} + \boldsymbol{h}) - f(\boldsymbol{p}) - \nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{h} = o(||\boldsymbol{h}||) \qquad \text{for} \qquad \boldsymbol{h} \to \boldsymbol{0}\]
This implies that
\[f(\boldsymbol{p} + h\boldsymbol{r}) - f(\boldsymbol{p}) - \nabla f(\boldsymbol{p})^{\mathsf{T}}(h\boldsymbol{r}) = o(h||\boldsymbol{r}||) \qquad \text{for} \qquad h \to 0\]
and so
\[f(\boldsymbol{p} + h\boldsymbol{r}) - f(\boldsymbol{p}) - h\nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{r} = ||\boldsymbol{r}|| o(h) \qquad \text{for} \qquad h \to 0\]
and
\[f(\boldsymbol{p} + h\boldsymbol{r}) - f(\boldsymbol{p}) - h\nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{r} = o(h) \qquad \text{for} \qquad h \to 0.\]
This implies the following:
\[\lim_{h \to 0} \frac{f(\boldsymbol{p} + h\boldsymbol{r}) - f(\boldsymbol{p}) - h\nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{r}}{h} = 0\]
We have:
\[\begin{aligned}\partial_{\boldsymbol{r}} f(\boldsymbol{p}) & = \lim_{h \to 0} \frac{f(\boldsymbol{p} + h\boldsymbol{r}) - f(\boldsymbol{p})}{h} \\ & = \lim_{h \to 0} \left(\frac{f(\boldsymbol{p} + h\boldsymbol{r}) - f(\boldsymbol{p}) - h\nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{r}}{h} + \frac{h\nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{r}}{h}\right) \\ & = \lim_{h \to 0} \frac{f(\boldsymbol{p} + h\boldsymbol{r}) - f(\boldsymbol{p}) - h\nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{r}}{h} + \lim_{h \to 0} \frac{h\nabla f(\boldsymbol{p})^{\mathsf{T}}\boldsymbol{r}}{h} \\ & = 0 + \nabla f(\boldsymbol{p})^{\mathsf{T}} \boldsymbol{r} \\ & = \nabla f(\boldsymbol{p})^{\mathsf{T}} \boldsymbol{r} \\ & = \nabla f(\boldsymbol{p}) \cdot \boldsymbol{r}\end{aligned}\]