Discrete-time, causal wiener filter

Stochastic $L_2$ optimal (least squares) FIR filter design problem: Given a wide-sense stationary (WSS) input signal $x_k$ and desired signal $d_k$ (WSS $(y_k)=(y_{k+d})$ , $r_{\mathrm{yz}}(l)=(y_{k}z_{k+l})$ , $\forall k, l\colon r_{\mathrm{yy}}(0)$∞ )

As posed, this problem seems slightly silly, since $d_k$ is already available! However, this idea is useful in a wide cariety of applications.

Determining the optimal length-n causal fir weiner filter

$$y_k=\sum_{l=0}^{M-1} w_l{x}_{k-l}$$ $$(, (^2))=((d_k-y_k)^2)=((d_k-\sum_{l=0}^{M-1} w_l{x}_{k-l})^2)=(d_k^2)-2\sum_{l=0}^{M-1} w_l(d_k{x}_{k-l})+\sum_{l=0}^{M-1} \sum_{m=0}^{M-1} (w_l{w}_m(x_{k-l}{x}_{k-m}))()$$ $$(^2)=r_{\mathrm{dd}}(0)-2\sum_{l=0}^{M-1} w_l{r}_{\mathrm{dx}}(l)+\sum_{l=0}^{M-1} \sum_{m=0}^{M-1} w_l{w}_m{r}_{\mathrm{xx}}(l-m)$$ where $$r_{\mathrm{dd}}(0)=(d_k^2)$$ $$r_{\mathrm{dx}}(l)=(d_k{X}_{k-l})$$ $$r_{\mathrm{xx}}(l-m)=(x_k{x}_{k+l-m})$$ This can be written in matrix form as $$(^2)=r_{\mathrm{dd}}(0)-2PW^T+W^TRW$$ where $$P=\begin{pmatrix}{r}_{\mathrm{dx}}(0)\\ r_{\mathrm{dx}}(1)\\ \\ r_{\mathrm{dx}}(M-1)\\ \end{pmatrix}$$ $$R=\begin{pmatrix}{r}_{\mathrm{xx}}(0) & r_{\mathrm{xx}}(1) & & & r_{\mathrm{xx}}(M-1)\\ r_{\mathrm{xx}}(1) & r_{\mathrm{xx}}(0) & & & \\ & & & & \\ & & & r_{\mathrm{xx}}(0) & r_{\mathrm{xx}}(1)\\ r_{\mathrm{xx}}(M-1) & & & r_{\mathrm{xx}}(1) & r_{\mathrm{xx}}(0)\\ \end{pmatrix}$$ To solve for the optimum filter, compute the gradient with respect to the top weights vector $W$ $$(, \begin{pmatrix}\frac{\partial^1^2}{\partial w_0}\\ \frac{\partial^1^2}{\partial w_1}\\ \\ \frac{\partial^1^2}{\partial w_{M-1}}\\ \end{pmatrix})$$ $$=-(2P)+2RW$$ (recall $\frac{d A^TW}{d W}=A^T$ , $\frac{d WMW}{d W}=2MW$ for symmetric $M$ ) setting the gradient equal to zero $$(W_{\mathrm{opt}}R=P, W_{\mathrm{opt}}=R^{(-1)}P)$$ Since $R$ is a correlation matrix, it must be non-negative definite, so this is a minimizer. For $R$ positive definite, the minimizer is unique.