From charlesreid1


This page covers mean and variance definitions for continuous random variables (with prescribed probability density function) and discrete random variables (with prescribed probability mass function).


Continuous Random Variables

If we have a continuous random variable X with a probability density function f(x), the mean and variance are given by:

\mu = E[X] = \int x f(x) dx

(where the integral is over the range of x values)

Discrete Random Variable

The mean of a discrete random variable X with discrete values x_i, 1 \leq i \leq n and a probability mass function p_i is given by the expression:

\mu = E[X] = \sum_{i=1}^{n} p_i x_i

Note that by definition, the probability mass function must sum to 1:

\sum_{i=1}^{n} p_i = 1

If we assume a uniform probability for each value, then the probability mass function of component i is just:

p_i = \frac{1}{n}


Continuous Random Variable

The variance of a continuous random variable is given by:

\sigma^2 = Var(X) = \int (x - \mu)^2 f(x) dx

Note that this can be expanded and simplified,

Var(X) = \int (x^2 - 2 x \mu + \mu^2) f(x) dx = \int x^2 f(x) d - 2 \mu \int x f(x) dx + \mu^2 \int f(x) dx = \int x^2 f(x) d - 2 \mu^2 + \mu^2 = \int x^2 f(x) dx - \mu^2

or just

\sigma^2 = Var(X) = \int x^2 f(x) dx - \mu^2

which is equivalent to saying:

Var(X) = E[X^2] - E[X]^2

Discrete Random Variable

If we have a discrete random variable with a probability mass function, the variance is given by:

\sigma^2 = Var(X) = \sum p_i (x_i - mu)^2 dx

This can be simplified to:

Var(X) = \sum_{i=1}^{n} p_i x_i^2 - \left( \sum_{i=1}^{n} p_i x_i \right)^2

where the last term is equal to the mean,

Var(X) = \sum_{i=1}^{n} p_i x_i^2 - \mu

Updating On The Fly

Updating Discrete Mean

Let's suppose we have n data values, x_i, 1 \leq i \leq n, and we are adding one additional data value x_{n+1} and want to compute the effect it has on the mean. For the sake of simplicity we assume that the probability mass function is uniform, so that p_i = \frac{1}{n} or p_i = \frac{1}{n+1}. Then old mean is given by:

\mu_{old} = \frac{1}{n} \sum_{i=1}^{n} x_i

and the new mean is given by:

\mu_{new} = \frac{1}{n+1} \sum_{i=1}^{n+1} x_i

We can re-express the new mean in terms of the old mean by the following relationship:

\mu_{new} = \left( \frac{n}{n+1} \right) \mu_{old} + \left( \frac{1}{n+1} \right) x_{n+1}

Updating Discrete Variance

Supposing the same situation - that we have n discrete data values x_i, 1 \leq i \leq n, and we are adding one additional data value x_{n+1} and want to compute the effect it has on the variance. As with updating the mean, we presume a uniform mass density function for the sake of simplicity, p_i = \frac{1}{n} or p_i = \frac{1}{n+1}. Then the old variance is given by:

Var_{old}(X) = \sum_{i=1}^{n} \left( \frac{x_i^2}{n} \right) - \left( \sum_{i=1}^{n} \frac{x_i}{n} \right)^2


Var_{old}(X) = \frac{1}{n} \sum_{i=1}^{n} x_i^2 - \mu_{old}^2

(as before, the last term is the old mean; because the probability mass function must sum to 1, the sum of 1/n is the same as the sum of 1/n^2, so the mismatch in the two terms is not a problem). Grouping the first term as SS (sum of squares) for convenience,

Var_{old}(X) = \frac{1}{n} SS_{old} - \mu_{old}^2

Now we can write out the new expression for the variance as:

Var_{new}(X) = \frac{1}{n+1} \sum_{i=1}^{n+1} \left( x_i^2 \right) - \mu_{new}^2

which can be written in terms of the old sum of squares term as:

Var_{new}(X) = \frac{n}{n+1} SS_{old} + \frac{1}{n} x_{n+1}^2 - \mu_{new}^2

Thus, if we want to start with a set of n points, and quickly evaluate the effect that adding one additional point would have (say, from a pool of possible "next choices"), we can implement the above formulas for a one-shot update.

See also: