Mon 23 April 2018

Notes on Covariance

Written by Hongjinn Park in Articles

It's helpful for me to keep in mind that Variance and Covariance are just Expectations or weighted averages to help describe a distribution.

Definition first, which I've found appears naturally when you calculate $Var(X+Y) = E[(X+Y - \mu_X -\mu_Y)^2]$

$$Cov(X,Y) = E[(X- E[X])(Y - E[Y])]$$

And by expanding the RHS we see that:

$$Cov(X,Y) = E[XY] - E[X]E[Y]$$

Note: if $X$ and $Y$ are independent, $E[g(X)h(Y)] = E[g(X)]E[h(Y)]$ and so $Cov(X,Y) = 0$.

Independence $\Rightarrow Cov(X,Y) = 0$ always, however the converse is not true!

$Cov(X,Y) = 0$ does not always mean independence! For example, $X$ is a RV such that $P(X=0) = P(X=1) = P(X=-1) = \frac{1}{3}$ and $Y = 0$ if $X \neq 0$ and $Y=1$ if $X=0$. Clearly $X$ and $Y$ are dependent but $E[XY] = E[X] = 0$ and so $Cov(X,Y) = 0$.

Properties

$$Cov(X,Y) = Cov(Y,X)$$ $$Cov(X,X) = Var(X)$$ $$Cov(aX+j,bY+k) = ab Cov(X,Y)$$ $$Cov(aX+bY,cW+dV) = ac Cov(X,W) + ad Cov(X,V) + bc Cov(Y,W) + bd Cov(Y,V) $$ $$Cov\bigg(\sum_{i=1}^{n}X_i,\sum_{j=1}^{m}Y_j\bigg) = \sum_{i=1}^{n}\sum_{j=1}^{m}Cov(X_i,Y_j)$$ $$Cov(X_1 + X_2, Y_1 + Y_2 + Y_3) = Cov(X_1,Y_1) + Cov(X_1,Y_2) + Cov(X_1,Y_3) + Cov(X_2,Y_1) + Cov(X_2,Y_2) + Cov(X_2,Y_3)$$ $$Var\bigg(\sum_{i=1}^{n}X_i\bigg) = \sum_{i=1}^{n} Var(X_i) + 2 \sum_{i < j } Cov(X_i,X_j)$$ Also since $Var(X_1+X_2)=Cov(X_1+X_2,X_1+X_2)$, $$Var\bigg(\sum_{i=1}^{n}X_i\bigg) = Cov\bigg(\sum_{i=1}^{n}X_i,\sum_{j=1}^{n}X_j\bigg) = \sum_{i=1}^{n}\sum_{j=1}^{n}Cov(X_i,X_j) $$ $$Var(X+Y+Z) = Var(X) + Var(Y) + Var(Z) + 2\bigg(Cov(X,Y) + Cov(X,Z) + Cov(Y,Z)\bigg)$$


Articles

Personal notes I've written over the years.