Introduction

Introduction

The tools, ideas, and insights from linear algebra, abstract algebra, and functional analysis can be extremely useful to signal processing and system theory in various areasof engineering, science, and social science. Indeed, many important ideas can be developed from the simple operator equation

by considering it in a variety of ways. If $\mathbf{x}$ and $\mathbf{b}$ are vectors from the same or, perhaps, different vector spaces and $\mathbf{A}$ is an operator, there are three interesting questions that can be asked which provide a setting for a broad study.

1. Given $\mathbf{A}$ and $\mathbf{x}$ , find $\mathbf{b}$ . The analysis or operator problem or transform.
2. Given $\mathbf{A}$ and $\mathbf{b}$ , find $\mathbf{x}$ . The inverse or control problem or deconvolution or design.
3. Given $\mathbf{x}$ and $\mathbf{b}$ , find $\mathbf{A}$ . The synthesis or design problem or parameter identification.

Much can be learned by studying each of these problems in some detail. We will generally look at the finite dimensional problem where [link] can more easily be studied as a finite matrix multiplication [link] , [link] , [link] , [link]

but will also try to indicate what the infinite dimensional case might be [link] , [link] , [link] , [link] .

An application to signal theory is in [link] , to optimization [link] , and multiscale system theory [link] . The inverse problem (number 2 above) is the basis for a large study of pseudoinverses, approximation, optimization, filter design, andmany applications. When used with the $l_2$ norm [link] , [link] powerful results can be optained analytically but used with other norms such as $l_{\infty }$ , $l_1$ , $l_0$ (a pseudonorm), an even larger set of problems can be posed and solved [link] , [link] .

A development of vector space ideas for the purpose of presenting wavelet representations is given in [link] , [link] . An interesting idea of unconditional bases is given by Donoho [link] .

Linear regression analysis can be posed in the form of [link] and [link] where the $\mathbf{M}$ rows of $\mathbf{A}$ are the vectors of input data from $\mathbf{M}$ experiments, entries of $\mathbf{x}$ are the $\mathbf{N}$ weights for the $\mathbf{N}$ components of the inputs, and the $\mathbf{M}$ values of $\mathbf{b}$ are the outputs [link] . This can be used in machine learning problems [link] , [link] . A problem similar to the design or synthesis problem is that of parameter identification where a model of somesystem is posed with unknown parameters. Then experiments with known inputs and measured outputs are run to identify these parameters. Linear regression is also anexample of this [link] , [link] .

Dynamic systems are often modelled by ordinary differential equation where $\mathbf{b}$ is set to be the time derivative of $\mathbf{x}$ to give what are called the linear state equations:

or for difference equations and discrete-time or digital signals,

which are used in digital signal processing and the analysis of certain algorithms. State equations are useful in feedback control as well as in simulation of manydynamical systems and the eigenvalues and other properties of the square matix $\mathbf{A}$ are important indicators of the performance [link] , [link] .

The ideas of similarity transformations, diagonalization, the eigenvalueproblem, Jordon normal form, singular value decomposition, etc. from linear algebra [link] , [link] , [link] are applicable to this problem.

Various areas in optimization and approximation use vector space math to great advantage [link] , [link] .

This booklet is intended to point out relationships, interpretations, and tools in linear algebra, matrix theory, and vector spaces that scientists and engineers might find useful. It is not astand-alone linear algebra book. Details, definitions, and formal proofs can be found in the references. A very helpful source is Wikipedia.

There is a variety software systems to both pose and solve linear algebra problems. A particularly powerful one is Matlab [link] which is, in some ways, the gold standard since it started years ago a purely numerical matrix package. Butthere are others such as Octave, SciLab, LabVIEW, Mathematica, Maple, etc.