Notes on Expected Value

---

Expected Value has many properties. I list some of them here.

It's definitely a good idea to understand Expected Value. Other concepts like Variance and Covariance are just Expected Values. You also find EV in the Law of Large Numbers and the Central Limit Theorem. In day to day life we commonly talk about EVs. There's also Moment Generating Functions, inequalities, predictions, etc.

Definitely an important concept. I should study it more.

---

For Discrete RV X,

$$E[X] = \sum_{x:p(x)>0}xp(x)$$ $$Var(X) = E[(X-\mu)^2] = \sum_{x}(x-\mu)^2 p(x) $$

For Continous RV X,

$$E[X] = \int_{-\infty}^{\infty} xf(x) dx$$ $$Var(X) = E[(X-\mu)^2] = \int_{-\infty}^{\infty} (x-\mu)^2 f(x) dx = \sigma^2$$

Properties for discrete or continuous

For ALL RVs $X$,

$$E[aX+b] = aE[X] + b$$ $$Var(aX+b) = a^2 Var(X)$$ $$Var(X) = E[X^2] - (E[X])^2$$ $$E[c] = c $$

For dependent or independent $X$ and $Y$ with finite $E[X]$ and $E[Y]$,

$$E[X+Y] = E[X] + E[Y]$$ $$E[aX + bY +c ] = aE[X] + bE[Y] + c$$

Dependent $X$ and $Y$,

$$Var(X+Y) = Var(X) + Var(Y) + 2Cov(X,Y)$$ $$E[c|X] = E[c] = c$$ $$E[X|c] = E[X]$$ $$E[X] = E[E[X|Y]] = \int_{-\infty}^{\infty} E[X|Y=y] f_Y(y) \,dy$$ $$Var(X|Y) = E[X^2|Y] - (E[X|Y])^2$$ $$Var(X) = E[Var(X|Y)] + Var(E[X|Y])$$ $$E[aX + bY | Z] = aE[X|Z] + bE[Y | Z] $$ $$E[X|X] = X \quad \text{also} \quad E[g(X)|X] = g(X)$$ $$Var(X|X) = 0 \quad \text{also} \quad Var(g(X)|X) = 0 $$ $$E[X|Y,g(Y)] = E[X|Y]$$

Note that $E[X|g(Y)]$ is not always equal to $E[X|Y]$, for instance let $Y=\pm 1$ with probability one half and let $g(y) = y^2$

$$E[X \mid Y, E[X|Y]] = E[X|Y] \quad \text{since $E[X|Y]$ is a function of $Y$}$$

Iterated Expectations

$$E[X | Y] = E[E[X | Z,Y] \mid Y]$$

For example, calculate the mean IQ of females by taking a probability weighted average of the mean IQ of female smokers and the mean IQ of female non smokers. For instance, find $E[IQ|Female]$ - Half of females are smokers with mean IQ 102 and the other half don't smoke with mean IQ 98, then the mean IQ of females is 100.

$$E[IQ \mid F] = E[IQ \mid S,F] P(S|F) + E[IQ \mid S^c,F] P(S^c| F) = E[E[IQ \mid F,S] \mid F]$$

This should remind you of $P(E|F) = P(E|GF)P(G|F) + P(E|G^cF) P(G^c|F)$

$$E[XY] = E[E[XY]|Y] = E[Y E[X|Y]] \quad \quad E[XY] = E[E[XY]|X] = E[X E[Y|X]]$$

Where given $Y$, $Y$ is like a constant that you can take out.

Always true:

• If $X \le Y$ then $E[X] \le E[Y]$ for any arbitrary RVs
• In all cases, $Var(X) \ge 0$ which also means $E[X^2] \ge (E[X])^2$
• $E[X|Y] \ge 0$ if $X \ge 0$
• If $P(a \le X \le b) = 1$ then $a \le E[X] \le b$
• $E[E[X]] = E[X]$
• $E[X-E[X]] = E[X-\mu] = E[X] - E[X] = 0$
• For discrete only: $P(X=c) = 1 \Leftrightarrow Var(X) = 0$
• For discrete only: $E[I_A] = P(A)$

INDEPENDENT ONLY, continuous or discrete

$X$ and $Y$ are independent with joint pdf $f_{X,Y}(x,y)$ or joint pmf $p_{X,Y}(x,y)$ then

$$E[g(X)h(Y)] = E[g(X)]E[h(Y)]$$

Also note: $E \left[ \frac{X}{Y} \right] = E[X]E \left[ \frac{1}{Y} \right]$ if $X$ and $Y$ are independent.

If $X$ and $Y$ independent,

$$Var(X+Y) = Var(X) + Var(Y)$$

Properties for Continuous only

For a non negative RV $Y$,

$$E[Y] = \int_{0}^{\infty} P\{Y > y\} dy$$

For example if $Y \sim Exp(\lambda)$ then, $E[Y] = \int_{0}^{\infty} e^{-\lambda y} dy = [-\frac{1}{\lambda} e^{-\lambda y}]_{0}^{\infty} = [0 - (-\frac{1}{\lambda})] = \frac{1}{\lambda}$

Functions of BOTH Continuous and Discrete RV X

Here $X$ is a RV and $g$ is any function,

$$E[g(X)] = \sum_{x} g(x)p(x) $$ $$E[g(X)] = \int_{-\infty}^{\infty} g(x)f(x) dx$$

Infinite collection of RVs

Linearity of Expectation holds true for an \textit{infinite} collection of RVs each having a finite expectation, if

1. Every RV in the infinite collection $X_i$ are non negative RVs

2. $\sum_{i=1}^{\infty} E[|X_i|] < \infty$

If these two conditions hold, then for an infinite collection:

$$E\Bigg[\sum_{i=1}^{\infty} X_i\Bigg] = \sum_{i=1}^{\infty} E[X_i]$$

Cool application of this, let $X_i = 1$ if $X \ge i$ and $0$ otherwise, then

$$E[X] = \sum_{i=1}^{\infty} E[X_i] = \sum_{i=1}^{\infty} P\{X \ge i\}$$

Examples:

a) $X$ is a constant RV equal to $100$ then $E[X]= (1)100 = 1+1+...+1 + 0 + 0+... = 100$

b) $X$ is a RV that is either $50$ or $100$ with equal probability. Then $E[X]= \frac{1}{2} 50 + \frac{1}{2}100 = 75 = 1+...+1 + .5+.5+...+.5 + 0 + 0+... = 50(1) + 50(.5) = 75$

c) $X$ is the outcome of dice rolls, then $E[X] = \frac{1}{6}(1+2+...+6) = 3.5 = 1 + \frac{5}{6} + \frac{4}{6} + ... + \frac{1}{6} + 0 +0 +... = 3.5$