5.4 Linear algebra: projections

Orthogonality. When the angle between two vectors is $\pi /2\left({90}^0\right)$ , we say that the vectors are orthogonal . A quick look at the definition of angle ( Equation 12 from "Linear Algebra: Direction Cosines" ) leads to this equivalent definition for orthogonality:

For example, in Figure 1(a) , the vectors $x=31$ and $y=-26$ are clearly orthogonal, and their inner product is zero:

In Figure 1(b) , the vectors $x=310,y=-260$ , and $z=004$ are mutually orthogonal, and the inner product between each pair is zero:

We can use the inner product to find the projection of one vector onto another as illustrated in Figure 2 . Geometrically we find the projection of $x$ onto $y$ by dropping a perpendicular from the head of $x$ onto the line containing $y$ . The perpendicular is the dashed line in the figure. The point where the perpendicular intersects $y$ (or an extension of $y$ ) is the projection of $x$ onto $y$ , or the component of $x$ along $y$ . Let's call it $z$ .

The vector $z$ lies along $y$ , so we may write it as the product of its norm $\left|\right|z\left|\right|$ and its direction vector $u_y$ :

But what is norm $\left|\right|z\left|\right|$ ? From Figure 2 we see that the vector $x$ is just $z$ , plus a vector $v$ that is orthogonal to $y$ :

Therefore we may write the inner product between $x$ and $y$ as

But because $z$ and $y$ both lie along $y$ , we may write the inner product $(x,y)$ as

From this equation we may solve for $\left|\right|z\left|\right|=\frac{(x,y)}{\left|\right|y\left|\right|}$ and substitute $\left|\right|z\left|\right|$ into Equation 6 to write $z$ as

Equation 10 is what we wanted–an expression for the projection of $x$ onto $y$ in terms of $x$ and $y$ .

Orthogonal Decomposition. You already know how to decompose a vector in terms of the unit coordinate vectors,

In this equation, $(x,e_k)e_k$ is the component of $x$ along $e_k$ , or the projection of $x$ onto $e_k$ , but the set of unit coordinate vectors is not the only possible basis for decomposing a vector. Let's consider an arbitrary pair of orthogonalvectors $x$ and $y$ :

The sum of $x$ and $y$ produces a new vector $w$ , illustrated in Figure 3 , where we have used a two-dimensional drawing to represent $n$ dimensions. The norm squared of $w$ is

${\left|\right|w\left|\right|}^2=(w,w)=[(x+y),(x+y)]=(x,x)+(x,y)+(y,x)+(y,y)={\left|\right|x\left|\right|}^2+{\left|\right|y\left|\right|}^2.$

This is the Pythagorean theorem in $n$ dimensions! The length squared of $w$ is just the sum of the squares of the lengths of its two orthogonal components.

The projection of $w$ onto $x$ is $x$ , and the projection of $w$ onto $y$ is $y$ :

If we scale $w$ by $a$ to produce the vector $z=aw$ , the orthogonal decomposition of $z$ is

Let's turn this argument around. Instead of building $w$ from orthogonal vectors $x$ and $y$ , let's begin with arbitrary $w$ and $x$ and see whether we can compute an orthogonal decomposition. The projection of $w$ onto $x$ is found from Equation 10 :

But there must be another component of $w$ such that $w$ is equal to the sum of the components. Let's call the unknown component $w_y$ . Then

Now, since we know $w$ and $w_x$ already, we find $w_y$ to be

Interestingly, the way we have decomposed $w$ will always produce $w_x$ and $w_{>y}$ orthogonal to each other. Let's check this:

To summarize, we have taken two arbitrary vectors, $w$ and $x$ , and decomposed $w$ into a component in the direction of $x$ and a component orthogonal to $x$ .