Inner Product Spaces

Definition: Inner Product Space

An inner product space is either a complex vector space $(V,\mathbb{C},+,\cdot )$ or a real vector space $(V,\mathbb{R},+,\cdot )$ equipped with an inner product operation $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{C}$ which has the following properties:

1. Conjugate symmetry

$$⟨\mathbf{u},\mathbf{v}⟩=\overline{⟨\mathbf{v},\mathbf{u}⟩}\forall \mathbf{u},\mathbf{v}\in V$$

2. Sesquilinearity

• Distributivity I: $⟨\mathbf{u}+\mathbf{v},\mathbf{w}⟩=⟨\mathbf{u},\mathbf{w}⟩+⟨\mathbf{v},\mathbf{w}⟩\forall \mathbf{u},\mathbf{v},\mathbf{w}\in V$

• Distributivity II: $⟨\mathbf{u},\mathbf{v}+\mathbf{w}⟩=⟨\mathbf{u},\mathbf{v}⟩+⟨\mathbf{u},\mathbf{w}⟩\forall \mathbf{u},\mathbf{v},\mathbf{w}\in V$

• Semi-linearity in the first argument: $⟨\lambda \mathbf{u},\mathbf{v}⟩=\bar{\lambda }⟨\mathbf{u},\mathbf{v}⟩\forall \mathbf{u},\mathbf{v}\in V,\forall \lambda \in \mathbb{C}$

• Linearity in the second argument: $⟨\mathbf{u},\lambda \mathbf{v}⟩=\lambda ⟨\mathbf{u},\mathbf{v}⟩\forall \mathbf{u},\mathbf{v}\in V,\forall \lambda \in \mathbb{C}$

3. Positive-definiteness

$$⟨\mathbf{v},\mathbf{v}⟩\ge 0\text{ with }⟨\mathbf{v},\mathbf{v}⟩=0⟺\mathbf{v}=0\forall \mathbf{v}\in V$$

NOTE

When we have a real inner product space, the inner product should be real-valued, i.e. $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{R}$. Conjugation is then simply ignored because it has no effect on real numbers.

NOTATION

We write $(V,F,+,\cdot )$ when it does not matter if the inner product space is real or complex.

Theorem: Euclidean Norm

Let $(V,F,+,\cdot )$ be an inner product space.

The function $||\cdot ||:V\to \mathbb{R}$ defined as

$$||\mathbf{v}||\stackrel{\text{def}}{=}\sqrt{⟨\mathbf{v},\mathbf{v}⟩}$$

is a norm on $(V,F,+,\cdot )$.

PROOF

TODO

Definition: Euclidean Norm

This norm is known as the Euclidean norm or the canonical norm on $(V,F,+,\cdot )$.

NOTATION

$$||\mathbf{v}|||\mathbf{v}|$$

Definition: Angle

The angle between two nonzero vectors $\mathbf{u}$ and $\mathbf{v}$ of an inner product space is defined using the Euclidean norm and the real arccosine function as follows:

$$\arccos \left(\frac{⟨\mathbf{u},\mathbf{v}⟩}{||\mathbf{u}||||\mathbf{v}||}\right)$$

NOTATION

$$\angle (\mathbf{u},\mathbf{v})$$

Theorem: Euclidean Metric

Let $(V,F,+,\cdot )$ be an inner product space.

The function $d:V\times V\to \mathbb{R}$ defined as

$$d(\mathbf{u},\mathbf{v})\stackrel{\text{def}}{=}||\mathbf{u}-\mathbf{v}||$$

is a metric on $V$.

PROOF

TODO

Definition: Euclidean Metric

We call $d$ the Euclidean metric on $V$.

Definition: Euclidean Distance

We call $d(\mathbf{u},\mathbf{v})$ the Euclidean distance between $\mathbf{u}$ and $\mathbf{v}$.

Orthogonality

Definition: Orthogonality

Two vectors $\mathbf{u}$ and $\mathbf{v}$ of an inner product space are orthogonal if their inner product is zero.

$$⟨\mathbf{u},\mathbf{v}⟩=0$$

NOTATION

$$\mathbf{u}\perp \mathbf{v}$$

Definition: Orthogonal Complement

Let $(U,F,+,\cdot )$ be a subspace of an [inner product space] $(V,F,+,\cdot )$.

The orthogonal complement of $U$ is the set of all vectors in $V$ which are orthogonal to all orthogonal in $U$.

$$\{\mathbf{v}\in V\mid \mathbf{v}\perp \mathbf{u},\forall \mathbf{u}\in U\}$$

NOTATION

$U^{\perp }$

THEOREM

The orthogonal complement of $(U^{\perp },F,+,\cdot )$ is also a subspace of $(V,F,+,\cdot )$ and its dimension is

$$\dim (U^{\perp })=\dim (V)-\dim (U).$$

PROOF

TODO

Definition: Orthogonal Basis

A basis $B$ of an inner product space $(V,F,+,\cdot )$ is orthogonal if all pairs of basis elements are orthogonal to each other:

$$\mathbf{u}\perp \mathbf{v}\forall \mathbf{u},\mathbf{v}\in B$$

Definition: Orthonormal Basis

An orthonormal basis of an inner product space is an orthogonal basis $B$, where the Euclidean norm of each basis element is equal to one:

$$||\mathbf{v}||=1\forall \mathbf{v}\in B$$

Algorithm: Gram-Schmidt Orthonormalisation Process

Every orthogonal basis $B=\{{\mathbf{b}}_1,\cdots ,{\mathbf{b}}_n\}$ of a finite-dimensional inner product space $(V,F,+,\cdot )$ can be turned into an orthonormal basis $B^{\prime }=\{{\mathbf{b}}_1^{\prime },\cdots ,{\mathbf{b}}_n^{\prime }\}$ through the following process:

1. The first vector ${\mathbf{b}}_1^{\prime }$ in $B^{\prime }$ is simply the normalized version of ${\mathbf{b}}_1$.

$${\mathbf{b}}_1^{\prime }\leftarrow \frac{1}{||{\mathbf{b}}_1||}{\mathbf{b}}_1$$

1. We construct the $k$-th element ${\mathbf{b}}_k^{\prime }$ of $B^{\prime }$ so that it is orthogonal to all the resulting basis vectors $\{{\mathbf{b}}_1^{\prime },\cdots ,{\mathbf{b}}_{k-1}^{\prime }\}$ before it and so that its norm is one.

• Firstly, we ensure that ${\mathbf{b}}_k^{\prime }$ is orthogonal to all $\{{\mathbf{b}}_1^{\prime },\cdots ,{\mathbf{b}}_{k-1}^{\prime }\}$ by assigning it the following value.

$${\mathbf{b}}_k^{\prime }\leftarrow {\mathbf{b}}_k-\sum _{i=1}^{k-1}⟨{\mathbf{b}}_k,{\mathbf{b}}_i^{\prime }⟩{\mathbf{b}}_i^{\prime }$$

• We then normalize ${\mathbf{b}}_k^{\prime }$ to ensure that its norm is one.

$${\mathbf{b}}_k^{\prime }\leftarrow \frac{1}{||{\mathbf{b}}_k^{\prime }||}{\mathbf{b}}_k^{\prime }$$

Theorem: Vector Representation through an Orthonormal Basis

Let $(V,F,+,\cdot )$ be an inner product space and let $\mathbf{v}\in V$.

If $B=\{{\mathbf{b}}_1,\cdots ,{\mathbf{b}}_n\}$ is an orthonormal basis, then the $i$-th coefficient $c_i$ in the basis representation $\mathbf{v}={\sum }_{i=1}^n{c}_i{\mathbf{b}}_i$ is the inner product of $\mathbf{v}$ with the $i$-th basis vector ${\mathbf{b}}_i$:

$$c_i=⟨\mathbf{v},{\mathbf{b}}_i⟩$$

PROOF

$$⟨\mathbf{v},{\mathbf{b}}_i⟩=c_1⟨{\mathbf{b}}_1,{\mathbf{b}}_i⟩+\cdots +c_i⟨{\mathbf{b}}_i,{\mathbf{b}}_i⟩+\cdots +c_n⟨{\mathbf{b}}_n,{\mathbf{b}}_i⟩$$

Since the basis is orthogonal, $⟨{\mathbf{b}}_j,{\mathbf{b}}_i⟩=0$ for all $j\ne i$. Moreover, the basis is orthonormal and so $⟨{\mathbf{b}}_i,{\mathbf{b}}_i⟩=||{\mathbf{b}}_i|{|}^2=1^2=1$ which means that

$$⟨\mathbf{v},{\mathbf{b}}_i⟩=c_1\underset{0}{\underbrace{⟨{\mathbf{b}}_1,{\mathbf{b}}_i⟩}}+\cdots +c_i\underset{1}{\underbrace{⟨{\mathbf{b}}_i,{\mathbf{b}}_i⟩}}+\cdots +c_n\underset{0}{\underbrace{⟨{\mathbf{b}}_n,{\mathbf{b}}_i⟩}}=c_i$$