12.1 Covariance and the correlation coefficient

Covariance and the correlation coefficient

The mean value ${\mu }_X=E\left[X\right]$ and the variance ${\sigma }_X^2=E\left[{(X-{\mu }_X)}^2\right]$ give important information about the distribution for real random variable X . Can the expectation of an appropriate function of $(X,Y)$ give useful information about the joint distribution? A clue to one possibility is given in the expression

The expression $E\left[XY\right]-E\left[X\right]E\left[Y\right]$ vanishes if the pair is independent (and in some other cases). We note also that for ${\mu }_X=E\left[X\right]$ and ${\mu }_Y=E\left[Y\right]$

To see this, expand the expression $(X-{\mu }_X)(Y-{\mu }_Y)$ and use linearity to get

which reduces directly to the desired expression. Now for given ω , $X\left(\omega \right)-{\mu }_X$ is the variation of X from its mean and $Y\left(\omega \right)-{\mu }_Y$ is the variation of Y from its mean. For this reason, the following terminology is used.

Definition . The quantity $\mathrm{Cov}[X,Y]=E\left[(X-{\mu }_X)(Y-{\mu }_Y)\right]$ is called the covariance of X and Y .

If we let $X^{\text{'}}=X-{\mu }_X$ and $Y^{\text{'}}=Y-{\mu }_Y$ be the centered random variables, then

Note that the variance of X is the covariance of X with itself.

If we standardize, with $X^{*}=(X-{\mu }_X)/{\sigma }_X$ and $Y^{*}=(Y-{\mu }_Y)/{\sigma }_Y$ , we have

Definition . The correlation coefficient $\rho =\rho [X,Y]$ is the quantity

Thus $\rho =\mathrm{Cov}[X,Y]/{\sigma }_X{\sigma }_Y$ . We examine these concepts for information on the joint distribution. By Schwarz' inequality (E15), we have

Now equality holds iff

We conclude $-1\le \rho \le 1$ , with $\rho =\pm 1$ iff $Y^{*}=\pm X^{*}$

Relationship between ρ and the joint distribution

• We consider first the distribution for the standardized pair $(X^{*},Y^{*})$
• Since ${\displaystyle P(X^{*}\le r,Y^{*}\le s)=P\left(\frac{X-{\mu }_X}{{\sigma }_X},\le ,r,,,,\frac{Y-{\mu }_Y}{{\sigma }_Y},\le ,s\right)}$
  $$=P(X\le t={\sigma }_{X}r+{\mu }_X,Y\le u={\sigma }_{Y}s+{\mu }_Y)$$
  we obtain the results for the distribution for $(X,Y)$ by the mapping
  $$\begin{array}{c}t={\sigma }_{X}r+{\mu }_X\hfill \\ u={\sigma }_{Y}s+{\mu }_Y\hfill \end{array}$$

Joint distribution for the standardized variables $(X^{*},Y^{*})$ , $(r,s)=(X^{*},Y^{*})\left(\omega \right)$

• $\rho =1$ iff $X^{*}=Y^{*}$ iff all probability mass is on the line $s=r$ .
• $\rho =-1$ iff $X^{*}=-Y^{*}$ iff all probability mass is on the line $s=-r$ .

If $-1<\rho <1$ , then at least some of the mass must fail to be on these lines.

The $\rho =\pm 1$ lines for the $(X,Y)$ distribution are:

Consider $Z=Y^{*}-X^{*}$ . Then ${\displaystyle E\left[\frac{1}{2}{Z}^2\right]=\frac{1}{2}E\left[{(Y^{*}-X^{*})}^2\right]}$ . Reference to [link] shows this is the average of the square of the distancesof the points $(r,s)=(X^{*},Y^{*})\left(\omega \right)$ from the line $s=r$ (i.e., the variance about the line $s=r$ ). Similarly for $W=Y^{*}+X^{*}$ , $E[W^2/2]$ is the variance about $s=-r$ . Now

Thus

• $1-\rho$ is the variance about $s=r$ (the $\rho =1$ line)
• $1+\rho$ is the variance about $s=-r$ (the $\rho =-1$ line)

Now since

the condition $\rho =0$ is the condition for equality of the two variances.

Transformation to the $(X,Y)$ plane

The $\rho =1$ line is: