Kind of embarrassing, but I'm completely blanking on what applying the expectation operator to a matrix means, and I can't find a simple explanation anywhere, or an example of how to carry out the computation. I'm learning about Wiener filtering, which can be used for a number of different purposes. Basically, the Wiener filter works as as mean-square-error minimization between the input signal and the "target" signal. The signal is assumed to be composed of our target and noise :
$$ x[n] = y[n] + w[n] $$
Where $x$ is the input signal, $y$ is the target and $w$ is noise. All are assumed to be wide-sense stationary random processes, meaning that the mean and covariance do not change in time.
Where I'm getting hung up is that the covariance of $x$ with respect to itself (the variance matrix) is calculated as
$$ C_{x} = E[xx^{T}] $$
but I don't remember what it means to apply the expectation operator to a matrix (in this case, $xx^{T}$).
EDIT:
Perhaps, I should be more clear in that what I don't understand is how to apply the expectation operator to the individual values within the matrix. For instance, the entry in the first row, first column of $$C_{x}$$ will be $$x[0] * x[0] = x[0]^{2}$$. How does one calculate $$E[x[0]^{2}]$$, and what does that mean?
$\endgroup$ 32 Answers
$\begingroup$The expected value of matrix $A$, $E(A)$, is just a matrix containing the expected values of the elements of the matrix $A$, as has already been mentioned.
Moreover, a number has no expected value, being a constant. The expression $E(XX^T)$ refers to the $X's$ viewed as random variables, and not as the specific realizations that you have available.
So the expression in the upper most left corner will be $$E(X_0X_0) = E(X_0^2)$$ but you are not supposed to insert in place of $X_0$ the actual value $x_0$ you have available as a realization of the random variable $X_0$. This is a theoretical expression.
If you want to obtain an estimate of these expected values, you have to invoke the covariance-stationarity assumption. When processes are stationary, any continuous transformation or combination of them is also stationary, and also, sample means are consistent estimators of theoretical moments.
Covariance stationarity implies $E(X_0^2) = E(X_1^2)=... =E(X_n^2)$
Define $Z_0= X_0^2$ etc. Then $E(Z_0) = E(Z_1^2)=... E(Z_n^2)$, and the $Z$'s are stationary. If they are stationary, the expression
$$\frac 1{n+1} \sum_{i=0}^{n}z_i \rightarrow_p E(Z_0)= E(X_0^2)$$
and the lowercase letters denote the actual realizations, the data sample you have available.
Moreover, covariance stationarity implies not necessarily that the covariances are constant through time, but that they only depend on the distance between two elements, and not on their exact position in the sequence (i.e. not on the time index). So a priori you assume that, for example, $E(X_0X_1) = E(X_1X_2) =...= E(X_{n-1}X_n)$. If you want to estimate this one-step covariance, by the same argument as before, you can calculate
$$\frac 1{n} \sum_{i=1}^{n}x_ix_{i-1} \rightarrow_p E(X_1X_2)$$
Note that the longer the gap between two variables, the more information from your sample you are going to lose in estimating their covariance.
Finally note that the expression $E(XX^T)$ is the (auto)covariance matrix only if the $X$'s have zero expected value.
$\endgroup$ 1 $\begingroup$$$ E[x x^T] = E[(y+n) (y+n)^T] = E[yy^T + y n^T + n y^T + nn^T]\\ = R_{yy} + R_{yn} + R_{ny} + R_{nn} \\= R_{yy} + R_{nn} ~~\text{ if $y$ and $n$ are uncorrelated.}$$
$\endgroup$ 1
