0.3 Low-dimensional signal models  (Page 3/3)

Manifold models

Manifold models generalize the conciseness of sparsity-based signal models. In particular, in many situations where a signal isbelieved to have a concise description or “few degrees of freedom,” the result is that the signal will live on or near aparticular submanifold of the ambient signal space.

Parametric models

We begin with an abstract motivation for the manifold perspective. Consider a signal $f$ (such as a natural image), and suppose that we can identify some single 1-D piece of information about thatsignal that could be variable; that is, other signals might rightly be called “similar” to $f$ if they differ only in this piece of information. (For example, this 1-D parameter coulddenote the distance from some object in an image to the camera.) We let $\theta$ denote the variable parameter and write the signal as $f_{\theta }$ to denote its dependence on $\theta$ . In a sense, $\theta$ is a single “degree of freedom” driving the generation of the signal $f_{\theta }$ under this simple model. We let $\Theta$ denote the set of possible values of the parameter $\theta$ . If the mapping between $\theta$ and $f_{\theta }$ is well-behaved, then thecollection of signals $\{f_{\theta }:\theta \in \Theta \}$ forms a 1-D path in the ambient signal space.

More generally, when a signal has $K$ degrees of freedom, we may model it as depending on some parameter $\theta$ that is chosen from a $K$ -dimensional manifold $\Theta$ . (The parameter space $\Theta$ could be, for example, a subset of ${\mathbb{R}}^K$ , or it could be a more general manifold such as SO(3).) We again let $f_{\theta }$ denote the signal corresponding to a particular choice of $\theta$ , and we let $\mathcal{F}=\{f_{\theta }:\theta \in \Theta \}$ . Assuming the mapping $f$ is continuous and injective over $\Theta$ (and its inverse is continuous), then by virtue of the manifold structure of $\Theta$ , its image $\mathcal{F}$ will correspond to a $K$ -dimensional manifold embedded in the ambient signal space (see [link] (c)).

These types of parametric models arise in a number of scenarios in signal processing. Examples include: signals of unknowntranslation, sinusoids of unknown frequency (across a continuum of possibilities), linear radar chirps described by a starting andending time and frequency, tomographic or light field images with articulated camera positions, robotic systems with few physicaldegrees of freedom, dynamical systems with low-dimensional attractors  [link] , [link] , and so on.

In general, parametric signals manifolds are nonlinear (by which we mean non-affine as well); this can again be seen byconsidering the sum of two signals $f_{{\theta }_0}+f_{{\theta }_1}$ . In many interesting situations, signal manifolds are non-differentiable as well.

Nonparametric models

Manifolds have also been used to model signals for which there is no known parametric model. Examples include images of faces andhandwritten digits [link] , [link] , which have been found empirically to cluster near low-dimensional manifolds.Intuitively, because of the configurations of human joints and muscles, it may be conceivable that there are relatively “few”degrees of freedom driving the appearance of a human face or the style of handwriting; however, this inclination is difficult orimpossible to make precise. Nonetheless, certain applications in face and handwriting recognition have benefitted from algorithmsdesigned to discover and exploit the nonlinear manifold-like structure of signal collections. Manifold Learning from Dimensionality Reduction discusses such methods for learning parametrizations and other information from data livingalong manifolds.

Much more generally, one may consider, for example, the set of all natural images. Clearly, this set has small volume with respect to the ambient signal space — generating an imagerandomly pixel-by-pixel will almost certainly produce an unnatural noise-like image. Again, it is conceivable that, at least locally,this set may have a low-dimensional manifold-like structure: from a given image, one may be able to identify only a limited numberof meaningful changes that could be performed while still preserving the natural look to the image. Arguably, most work insignal modeling could be interpreted in some way as a search for this overall structure.