1.5 Projection theorem

Uniqueness of decompositions

For the next property, we need two quick definitions.

Definition 1 The sum of two spaces $S_1,S_2\subseteq X$ is the set $S_1+S_2=\left\{x,+,y,:,x,\in ,S_1,,,y,\in ,S_2\right\}$ .

Definition 2 The subspaces $S_1,S_2\subseteq X$ are disjoint if $S_1\cap S_2=\left\{0\right\}$ .

Theorem 1 Let $V$ , $W\subseteq X$ be subspaces. For each $x\in V+W$ there exists a unique pair $v\in V,w\in W$ such that $x=v+w$ if and only if $V$ and $W$ are disjoint.

Proof: We prove the two directions of this “if and only if” statement separately.

$(\Rightarrow )$ If for each $x\in X$ there exists a unique pair $v,w\in V\times W$ such that $x=v+w$ , then the subspaces $V$ and $W$ are disjoint.

Assume there exists a unique pair $v,w$ s.t. $x=v+w$ for each $x\in V+W$ . For the sake of contradiction, assume $V$ and $W$ are not disjoint, i.e. there exists some $z\in V\cap W$ such that $z\ne 0$ . Pick $x$ and the corresponding $v,w$ such that $x=v+w$ . Then,

We examine these two terms in the equation:

Also note

So we have two pairs of elements from $V$ and $W$ which sum to $x$ , a contradiction to the assumption that these pairs are unique. Therefore, $V\cap W=\left\{0\right\}$ and thus $V$ and $W$ are disjoint.

$(\Leftarrow )$ If $V$ and $W$ are disjoint, then for each $x\in V+W$ there exists a unique pair $v,w\in V\times W$ such that $v+w=x$ .

To prove uniqueness, we will assume that two distinct pairs exist and show at the end that they are qual to each other. The two pairs we begin with are

so the pairs are distinct from each other. Subtract the equations:

We examine these two terms in the equation:

Since those two terms are equal,

which means that $w-q\in V\cap W$ and $p-v\in V\cap W$ . Our starting assumption was that $V\cap W=\left\{0\right\}$ , so therefore $w-q=0$ and $p-v=0$ which implies $w=q$ and $p=v$ . This statement is a contradiction since we assumed that the pairs were distinct, i.e., $w\ne q$ and $p\ne v$ . Therefore, only a unique pair $v\in V$ , $w\in W$ exists such that $v+w=x$ .

Fact 1 If $S$ is a subspace, then $S$ and $S^{\perp }$ are disjoint. To see this, assume $x\in S\cap S^{\perp }$ , which means $x\in S^{\perp }$ and so $⟨x,y⟩=0$ for all $y\in S$ . Pick $y=x$ so $⟨x,x⟩=0\Rightarrow x=0$ which means $S\cap S^{\perp }=\left\{0\right\}$ .

Fact 2 Using Fact 1, we can show that for each $x\in X$ there exists a unique pair $s\in S$ and $t\in S^{\perp }$ such that $x=s+t$ . In particular:

• If $x\in S$ , $s=x$ and $t=0$ .
• If $x\in S^{\perp }$ , $s=0$ and $t=x$ .
• If $x\notin S$ and $x\notin S^{\perp }$ , both $s,t\ne 0$ .

Projection theorem

In fact, there is quite a lot more we can say about the projection of a point $x\in X$ into a subspace $S\subseteq X$

Theorem 2 Let $X$ be a Hilbert space and $S$ be a closed subspace of $X$ . For any vector $x\in X$ there exists a unique point $s_0\in S$ that is closest to $x$ , i.e.,

In other words, $\parallel x-s_0\parallel \le \parallel x-s\parallel$ for all $s\in S$ with equality only if $s=s_0$ .

Furthermore, $s_0$ is a minimizer of $\parallel x-s\parallel$ over $s\in S$ if and only if $x-s_0\perp S$ . In other words, $x-s_0\in S^{\perp }$ .

Proof: To structure our proof, we will restate the theorem into four separate facts:

1. Existence: A minimizer of the distance $\parallel x-s\parallel$ exists in $S$ .
2. Orthogonality of the error: If $s_0$ is a minimizer, then $x-s_0\perp S$ .
3. Sufficiency of orthogonality: If $x-s_0\perp S$ , then $s_0$ is a minimizer to the distance function.
4. Uniqueness: only one point that minimizes the distance exists in $S$ .

Each of these facts is proven separately.

1. Existence: If $x\in S$ , then $s_0=x$ is the minimizer (since 0 is the minimum distance). If $x\notin S$ , we denote $\delta ={inf}_{s\in S}\parallel x-s\parallel$ . Note here that we use the infimum, which is the upper bound on all the lower bounds on the distance. The infimum is used because we do not yet know if there exists a point in $S$ that has lowest distance to $x$ . Note also that an infimum exists because norms have a lower bound.

We need to show that for some $s_0\in S$ , we have $\parallel x-s_0\parallel =\delta$ . Let $\left\{s_i\right\}\subseteq S$ be a sequence that yields $\parallel x-s_i\parallel \to \delta$ . We will show that this sequence is a Cauchy sequence; thenm using the fact that the space $S$ is closed and Hilbert, it is complete and therefore the Cauchy sequence must converge to a point $s_0\in S$ .

To prove that the sequence is Cauchy, we use the Parallelogram Law:

Since $S$ is a subspace, $\frac{s_i+s_j}{2}\in S$ so ${∥x,-,\frac{s_i+s_j}{2}∥}^2\ge {\delta }^2$ , since $\delta$ is the infimum. Therefore,

At this point, we note that $\parallel s_j-s_i\parallel \to 0$ as $i,j\to \infty$ , and so the sequence can be shown to be a Cauchy sequence. Since $\left\{s_i\right\}$ is Cauchy, it converges to a point $s_0$ and by the triangle inequality:

Now the first term of the inequality goes to $\delta$ as $j\to \infty$ and the second term goes to 0, and so we have that $\parallel x-s_0\parallel \le \delta +ϵ$ for all values of $ϵ>0$ . Since $\delta$ was the infimum of the norm on the left hand side, it follows that $\parallel x-s_0\parallel =\delta$ , and so we have found that a minimizer exists.

2. Orthogonality of the error: We proceed by contradiction by assuming $x-s_0$ is not orthonormal to $S$ , i.e., that there exists a unit-norm vector $\widehat{s}\in S$ such that $⟨x-s_0,\widehat{s}⟩=\delta \ne 0$ . Let $z=s_0+\delta \widehat{s}\in S$ . Then $\parallel x-s_0\parallel \le \parallel x-z\parallel$ . Furthermore,

Now we recognize two simplifications. First, we selected $\widehat{s}$ so that $\parallel \widehat{s}{\parallel }^2=1$ ; second, the remaining inner product is equal to $-2\text{Re}\left(\overline{\delta },⟨x-s_0,\delta \widehat{s}⟩\right)=-2\text{Re}\left(\overline{\delta }\delta \right)=-2{\left|\delta \right|}^2$ . Therefore,

This is a contradiction, since we have found a $z\in S$ closer to $x$ than $s_0$ . Thus $⟨x-s_0,\overline{s}⟩=0$ for all $\overline{s}\in S$ , which means that $x-s_0\perp S$ .

3. Sufficiency of orthogonality: Assume $x-s_0\perp S$ . For any $\tilde{s}\in S$ with $\tilde{s}\ne s_0$ , we have $\parallel x-\tilde{s}{\parallel }^2={\parallel x-s_0+s_0-\tilde{s}\parallel }^2$ . We use the Pythagorean theorem (if $a\perp b$ then ${\parallel a+b\parallel }^2={\parallel a\parallel }^2+{\parallel b\parallel }^2$ ): since $x-s_0\perp S$ and $s_0-\tilde{s}\in S$ , we have $\parallel x-\tilde{s}{\parallel }^2=\parallel x-s_0{\parallel }^2+{\parallel s_0-\tilde{s}\parallel }^2$ . Since $s_0\ne \tilde{s}$ , we have $\parallel s_0-\tilde{s}\parallel >0$ , and so $\parallel x-\tilde{s}{\parallel }^2>{\parallel x-s_0\parallel }^2$ . Now, since we picked $\tilde{s}\in S$ arbitrarily, it follows that $s_0$ is the minimizer of $\parallel x-s\parallel$ over $s\in S$ .

4. Uniqueness: We can write $x=x-s_0+s_0$ . From previous parts, we know that $x-s_0\in S^{\perp }$ and $s_0\in S$ . Since $S$ and $S^{\perp }$ are disjoint, only one pair of values $x-s_0$ and $x_0$ that holds these two properties exists. Therefore, the minimizer $s_0$ is unique.