Moment-Based Descriptors

Mean

• Expected value of random variable $X$, is a measure of the central tendency of the distribution of $X$. It is represented by $\mathbb{E}[X]$, and is the value we expect to obtain "on average" if we continue to take observations of $X$ and average out the results. The mean of a discrete distribution with PMF $p(x)$ is $$\mathbb{E}[X] = \sum_{x}xp(x)$$
• The mean of continuous random variable, with PDF $f(x)$ is $$\mathbb{E}[X] = \int_{-\infty}^{\infty}xf(x)dx$$
• Computing by Julia

using QuadGK

sup = (-1,1)
f1(x) = 3/4*(1-x^2)
f2(x) = x < 0 ? x+1 : 1-x

expect(f,support) = quadgk(x -> x*f(x), support...)[1]

println("Mean 1: ", expect(f1,sup))
println("Mean 2: ", expect(f2,sup))

General Expectation and Moments

• In general, for a function $h: \mathbb{R} \to \mathbb{R}$ and a random variable $X$, we can consider the random variable $Y=h(x)$. The distribution of $Y$ will typically be different from the distribution of $X$. $$\mathbb{E}[Y] = \mathbb{E}[h(X)] = \left\{\begin{split} & \sum_{x}h(x)p(x) & \text{for discrete}\\ & \int_{-\infty}^{\infty}h(x)f(x)dx \quad& \text{for continous} \end{split} \right.$$
• A common case is $h(x) = x^{\ell}$, in which case we call $\mathbb{E}(X\ell)$, the $\ell$-th moment of $X$. Then, for a random variable $X$ with PDF $f(x)$, the $\ell^{th}$ moment of $X$ is $$\mathbb{E}[X^{\ell}] = \int_{-\infty}^{\infty}x^{\ell}f(x)dx$$

Variance

• The variance of a random variable $X$, often denoted as $Var(X)$ or $\sigma^{2}$, is a measure of the spread, or dispersion, of the distribution of $X$. It is defined by $${\rm Var}(X) = \mathbb{E}[(X-\mathbb{E}[X])^{2}] = \mathbb{E}[X^{2}] - \big(\mathbb{E}[X]\big)^{2}$$
• Therefore, for discrete: $${\rm Var}(X) = \mathbb{E}[X^{2}] - \big(\mathbb{E}[X]\big)^{2} = \sum_{x}x^{2}p(x) - \bigg[\sum_{x}xp(x)\bigg]^{2}$$
• For continuous: $${\rm Var}(X) = \mathbb{E}[X^{2}] - \big(\mathbb{E}[X]\big)^{2} = \int_{-\infty}^{\infty}x^{2}f(x)dx - \bigg[\int_{-\infty}^{\infty}xf(x)dx\bigg]^{2}$$
• Besides, the variance of $X$ can also be considered as the expectation of a new random variable, $Y=(X-\mathbb{E}[X])^{2}$.
• Simulation by Julia $f(x) = \left\{\begin{array}{ll}x-4 &\quad\text{for }x\in [4,5],\\6-x&\quad\text{for }x \in (5,6].\end{array}\right.$

Higher Order Descriptors: Skewness and Kurtosis

• Take a random variable $X$ with $\mathbb{E}[X] = \mu$ and ${\rm Var}(X) = \sigma$, then the skewness is defined as $$\gamma_{3} = \mathbb{E}\bigg[\bigg(\frac{X-\mu}{\sigma}\bigg)^{3}\bigg] = \frac{\mathbb{E}[X^{3}] - 3\mu\sigma^{2} - \mu^{3}}{\sigma^{3}}$$
  • proof $$\begin{array}{ll} \because & \gamma_{3} = \mathbb{E}\big[\big(\frac{X-\mu}{\sigma}\big)^{3}\big] \\ \therefore & \gamma_{3} = \mathbb{E}\big[\frac{X^{3} - 3X^{2}\mu + 3\mu^{2}X - \mu^{3}}{\sigma^{3}}\big] = \frac{\mathbb{E}[X^{3}]-3\mu\mathbb{E}[X^{2}-X\mu]-\mu^{3}}{\sigma^{3}}\\ \because & 3\mu\mathbb{E}[X^{2} - X\mu] = 3\mu(\mathbb{E}[X]^{2}-\mu\mathbb{E}[X]) \text{ and } \mu=\mathbb{E}[X] \\ \therefore & 3\mu\mathbb{E}[X^{2} - X\mu] = 3\mu(\mathbb{E}[X^{2}]-(\mathbb{E}[X])^{2}) = 3\mu\sigma^{2} \\ \therefore & \gamma_{3} = \frac{\mathbb{E}[X^{3}] - 3\mu\sigma^{2} - \mu^{3}}{\sigma^{3}} \\ \end{array}$$
  • The skewness is a measure of the asymmetry of the distribution. For a distribution having a symmetric density function about the mean, we have $\gamma_{3} = 0$. Otherwise, it is either positive or negative depending on the distribution being skewed to the right or skewed to the left.
• The kurtosis is defined as $$\gamma_{4} = \mathbb{E}\bigg[\bigg(\frac{X-\mu}{\sigma}\bigg)^{4}\bigg] = \frac{\mathbb{E}[(X-\mu)^{4}]}{\sigma^{4}}$$
  • The kurtosis is a measure of the tails of the distribution. Any normal probability distribution has $\gamma_{4} = 3$. Then, a probability distribution with a higher value of $\gamma_{4}$ can be interpreted as having "heavier tails", while a probability distribution with a lower value is said to have "lighter tails".
  • Excess kurtosis defined as $\gamma_{4} - 3$
• $\gamma_{3}$ and $\gamma_{4}$ are invariant to changes in location and scale of the distribution.

Laws of Large Numbers

• Emperical averages converge to expected values. A sequence of independent and identically distributed random variables $X_{1}, X_{2},\cdots$ is considered. Then for each $n$, we compute the sample mean. $$\bar{X}_{n} = \frac{1}{n}\sum_{k=1}^{n}X_{k}$$
• If the expected value of each of the random variables $X_{i}$ is $\mu$, then a law of large numbers is a claim that the sequence $\{\bar{X}_{n}\}^{\infty}_{n=1}$ converge to $\mu$. The distinction between "weaks" and "strong" lies with the mode of converge.
  • The wake law of large numbers claims that the sequence of probabilities. As the $n$ grows, the likelihood of the sample mean $\bar{X}_{n}$ to be farther away than $\epsilon$ from the mean $\mu$ vanishes. $$w_{n} = \mathbb{P}(\vert \bar{X}_{n}-\mu\vert > \epsilon) \\\ \\ \lim_{n\to \infty}w_{n} = 0$$
  • The strong law of large numbers stats that $$\mathbb{P}\bigg(\lim_{n\to\infty} \bar{X}_{n} =\mu\bigg) = 1$$