<< Chapter < Page Chapter >> Page >
x , y 1 x , y 2 x , y n β = y 1 , y 1 y 2 , y 1 y n , y 1 y 1 , y 2 y 2 , y 3 y n , y 2 y 1 , y n y 2 , y n y n , y n G a 1 a 2 a n a .

The coefficients of the best approximation can then be obtained as a vector a = G - 1 β , as long as the Gram matrix G is invertible, i.e., it has a nonzero determinant.

In the case that x and y i are complex-valued vectors, one can rewrite the approximation i = 1 n a i y i = Y a , where a is the coefficient vector denoted above and Y = [ y 1 ... y n ] is a matrix that collects the vectors y i as its columns. The projection theorem requirement then becomes x - M a , y j = 0 for j = 1 ... , n , which can be rewritten as y j H ( x - M a ) = 0 and collected as before into the matrix equation

M j H ( x - M a ) = 0 , M H x - M H M a = 0 , M H M a = M H x , a = ( M H M ) - 1 M H x = M x ,

which is known as the least squares solution and exists as long as M H M is an invertible matrix. Once these coefficients are obtained, the approximation is equal to x ^ = M a = M ( M H M ) - 1 M H x ; therefore, the matrix P M = M ( M H M ) - 1 M H is known as the projection operator.

Application: channel equalization

We consider a linear channel with impulse response h that maps an input x into an output y :

x h y .

We wish to design a linear equalizer of impulse response g for the input x so that after it is run through the equalizer and the channel of impulse response h the output f x (i.e., f is as close as possible to x ):

x g h f .

Since the equalizer is linear, the order of g and h can be reversed (this will be discussed in more detail later):

x h g f .

Our design for g will be a finite impulse response filter with tap coefficients g i ; the mapping from input to output index n is therefore given by

f n = i = 1 k g i y n - i .

The error in approximating x n is given by

e n = f n - x n = i = 1 k g i y n - i - x n .

The total error magnitude over N observations is given by

E = i = 0 N - 1 e n 2 = i = 0 N - 1 i = 1 n ( g i y n - i - x n ) 2 .

We want to pose this question in terms of error of approximation into a subspace:

E = M g - b 2 2 .

By convention, we assume that the values g n = 0 and x n = 0 for n < 0 (i.e., n = 0 is the time of the first observation). It can be easily seen that

g = g 1 g k and b = x 1 x N ;

formulating M requires a separate study of the sum in [link] for each value of n . For n = 0 ,

i = 1 k g i y n - i = i = 1 k g i y - i = 0 ,

and so terms n 0 can be ignored. For n = 1 ,

i = 1 k g i y n - 1 = i = 1 k g i y 1 - i = [ y 0 , y - 1 , . . . , y - k + 1 ] g 1 g k = [ y 0 , 0 , . . . , 0 ] g 1 g k .

Similarly, for n = 2 ,

i = 1 k g i y n - 1 = i = 1 k g i y 2 - i = [ y 1 , y 0 , 0 , . . . , 0 ] g 1 g k .

Continuing until n = N ,

i = 1 k g i y n - 1 = i = 1 k g i y N - i = [ y N - 1 , y N - 2 , . . . , y N - k ] g 1 g k .

The concatenation of these sums as a vector can then be expressed by the matrix-vector product M g , where the matrix M is given by

M = y 0 0 0 0 0 y 1 y 0 0 0 0 y 2 y 1 y 0 0 0 y N - 1 y N - 2 y N - 3 y N - 4 y N - k .

Note that for M to have linearly independent columns (a condition for uniqueness of the solution to g ) the number of nonzero values of y i must be at least k . In this case, the solution

g = M b = ( M T M ) - 1 M T b

minimizes the error as established in the Projection Theorem.

Application: linear regression

In linear regression, we are given a set of input/output pairs ( x i , y i ) , i = 1 , ... , N , and we wish to find a linear relationship between inputs and outputs y i = a x i + b that minimize the sum of squared errors E = i = 1 N ( y i - ( a x i + b ) 2 ) . As in previous examples, we seek to pose this minimization problem in terms of the problem considered by the projection theorem: the error E = M g - y 2 2 , where M is a matrix, y is a vector, and g is the optimization variable vector. One can easily see that the following choice achieves the desired equality:

M = x 1 1 x 2 1 x N 1 , g = a b , and y = y 1 y N .

As before, the solution that minimizes the error is given by

g = ( M T M ) - 1 M T y ,

which exists and is unique as long as G = M T M is invertible, i.e., as long as M has linearly independent columns, i.e., as long as not all x i are equal. Now, we see that

M T M = x 1 x n 1 1 x 1 1 x n 1 = i = 1 N x i 2 i = 1 n x i i = 1 N x i N , ( M T M ) - 1 = N - i = 1 N x i - i = 1 N x i i = 1 N x i 2 N i = 1 N x i 2 - ( i = 1 n x i ) 2 , M T y = i = 1 N x i y i i = 1 n y i .

Collecting these results, we have that

g = N i = 1 n x i y i - ( i = 1 N x i ) ( i = 1 n y i ) ( i = 1 N x i 2 ) ( i = 1 N y i ) - ( i = 1 N x i ) ( i = 1 N x i y i ) N i = 1 N x i 2 - ( i = 1 N x i ) 2 .

Least squares: rejoinder

We have studied several examples where an optimization problem can be formulated as

g ^ = arg min g M g - b 2 2 ,

where M is a matrix and b and g are column vectors of appropriate sizes.

  • When the columns of M are orthonormal (i.e., orthogonal and unit norm), we compute g ^ = M H b , i.e., g ^ i = M i , b , where g ^ i is the i t h entry of g and M i is the i t h column of M .
  • When the columns of M are linearly independent, we compute g ^ = M b = ( M H M ) - 1 M H b . The Moore-Penrose pseudoinverse M = ( M H M ) - 1 M H is well defined because the Gram matrix G = M H M of a linearly independent set is invertible.
  • When the columns of M are linearly dependent, there exists a vector a 0 such that M a = 0 . Thus, if a solution g 0 to the optimization exists, then g 0 + a is also a solution: note that M g 0 - b = M ( g 0 + a ) - b , and so we lose uniqueness of the minimizer; in fact, we will have infinitely many solutions to the minimization. However, there are ways to rank the solutions and pick a "favorite", e.g., the solution with the smallest norm. This will be considered later in the course.

Questions & Answers

what is mutation
Janga Reply
what is a cell
Sifune Reply
how is urine form
Sifune
what is antagonism?
mahase Reply
classification of plants, gymnosperm features.
Linsy Reply
what is the features of gymnosperm
Linsy
how many types of solid did we have
Samuel Reply
what is an ionic bond
Samuel
What is Atoms
Daprince Reply
what is fallopian tube
Merolyn
what is bladder
Merolyn
what's bulbourethral gland
Eduek Reply
urine is formed in the nephron of the renal medulla in the kidney. It starts from filtration, then selective reabsorption and finally secretion
onuoha Reply
State the evolution relation and relevance between endoplasmic reticulum and cytoskeleton as it relates to cell.
Jeremiah
what is heart
Konadu Reply
how is urine formed in human
Konadu
how is urine formed in human
Rahma
what is the diference between a cavity and a canal
Pelagie Reply
what is the causative agent of malaria
Diamond
malaria is caused by an insect called mosquito.
Naomi
Malaria is cause by female anopheles mosquito
Isaac
Malaria is caused by plasmodium Female anopheles mosquitoe is d carrier
Olalekan
a canal is more needed in a root but a cavity is a bad effect
Commander
what are pathogens
Don Reply
In biology, a pathogen (Greek: πάθος pathos "suffering", "passion" and -γενής -genēs "producer of") in the oldest and broadest sense, is anything that can produce disease. A pathogen may also be referred to as an infectious agent, or simply a germ. The term pathogen came into use in the 1880s.[1][2
Zainab
A virus
Commander
Definition of respiration
Muhsin Reply
respiration is the process in which we breath in oxygen and breath out carbon dioxide
Achor
how are lungs work
Commander
where does digestion begins
Achiri Reply
in the mouth
EZEKIEL
what are the functions of follicle stimulating harmones?
Rashima Reply
stimulates the follicle to release the mature ovum into the oviduct
Davonte
what are the functions of Endocrine and pituitary gland
Chinaza
endocrine secrete hormone and regulate body process
Achor
while pituitary gland is an example of endocrine system and it's found in the Brain
Achor
what's biology?
Egbodo Reply
Biology is the study of living organisms, divided into many specialized field that cover their morphology, physiology,anatomy, behaviour,origin and distribution.
Lisah
biology is the study of life.
Alfreda
Biology is the study of how living organisms live and survive in a specific environment
Sifune
Got questions? Join the online conversation and get instant answers!
Jobilize.com Reply

Get Jobilize Job Search Mobile App in your pocket Now!

Get it on Google Play Download on the App Store Now




Source:  OpenStax, Signal theory. OpenStax CNX. Oct 18, 2013 Download for free at http://legacy.cnx.org/content/col11542/1.3
Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Signal theory' conversation and receive update notifications?

Ask