Standard deviation

In statistics, the standard deviation is a measure of the amount of variation of the values of a variable about its (arithmetic) average. A low standard deviation indicates that the values of a set tend to be close to their average, while a high standard deviation indicates that the values are spread out over a wider range. Standard deviation may be abbreviated SD or std dev, and is most commonly represented in mathematical texts and equations by the lowercase Greek letter σ (sigma).

In the case of a normal distribution (as typified by the symmetrical bell-shaped curve), approximately 68.3% of all values fall within one standard deviation of the average, 95.4% within two standard deviations, and 99.7% within three.

The standard deviation of a random variable, sample, statistical population, data set or probability distribution is the square root of its variance (the variance being the average of the squared deviations from the mean). A useful property of the standard deviation is that, unlike the variance, it is expressed in the same unit as the data. Standard deviation can also be used to calculate standard error for a finite sample, and to determine statistical significance.

When only a sample of data from a population is available, the term standard deviation of the sample or sample standard deviation can refer to either the above-mentioned quantity as applied to those data, or to a modified quantity that is an unbiased estimate of the population standard deviation (the standard deviation of the entire population).

The standard deviation of a population or sample and the standard error of a statistic (e.g., of the sample mean) are quite different, but related. The sample mean's standard error is the standard deviation of the set of means that would be found by drawing an infinite number of repeated samples from the population and computing a mean for each sample. The mean's standard error turns out to equal the population standard deviation divided by the square root of the sample size, and is estimated by using the sample standard deviation divided by the square root of the sample size. For example, a poll's standard error (what is reported as the margin of error of the poll) is the expected standard deviation of the estimated mean if the same poll were to be conducted multiple times. Thus, the standard error estimates the standard deviation of an estimate, which itself measures how much the estimate depends on the particular sample that was taken from the population.

In science, it is common to report both the standard deviation of the data (as a summary statistic) and the standard error of the estimate (as a measure of potential error in the findings). By convention, only effects more than two standard errors away from a null expectation are considered "statistically significant", a safeguard against spurious conclusion that is really due to random sampling error.

Suppose that the entire population of interest is eight students in a particular class. Their marks are the following eight values: \[2,\ 4,\ 4,\ 4,\ 5,\ 5,\ 7,\ 9.\]

For a finite set of numbers, the population standard deviation is found by taking the square root of the average of the squared deviations of the values subtracted from their average value, that is: \[\sigma = \sqrt{ \mathrm{average}\left((v - \mu)^2 \text{ for }v\in\mathrm{values}\right) } \text{ where } \mu = \mathrm{average}(\mathrm{values}).\]

These eight data points have the mean (average) of 5:\[\mu = \frac{2 + 4 + 4 + 4 + 5 + 5 + 7 + 9}{8} = \frac{40}{8} = 5.\]

Next, calculate the deviations of each data point from the mean. Then square the result of each:\[\begin{array}{lll} (2 - 5)^2 = (-3)^2 = 9 && (5 - 5)^2 = 0^2 = 0 \\ (4 - 5)^2 = (-1)^2 = 1 && (5 - 5)^2 = 0^2 = 0 \\ (4 - 5)^2 = (-1)^2 = 1 && (7 - 5)^2 = 2^2 = 4 \\ (4 - 5)^2 = (-1)^2 = 1 && (9 - 5)^2 = 4^2 = 16. \\ \end{array}\]

Next, the variance is the mean of these values: \[\sigma^2 = \frac{9 + 1 + 1 + 1 + 0 + 0 + 4 + 16}{8} = \frac{32}{8} = 4\]

Finally, the population standard deviation is the square root of the variance:\[\sigma = \sqrt{ 4 } = 2.\]

If the eight values with which we began are not the entire population of interest, then this is only one of many possible estimates of the standard deviation in the whole population. This specific one is biased, meaning that its expected value in repeated sampling deviates from the true value, but it is still consistent. Its mean squared error, on the other hand, may be lower than that of the unbiased estimator.

If the population of interest is approximately normally distributed, the standard deviation provides information on the proportion of observations above or below certain values. For example, the average height for adult men in the United States is about 69 inches, with a standard deviation of around 3 inches. This means that most men (about 68%, assuming a normal distribution) have a height within 3 inches of the mean (66-72 inches) – one standard deviation – and almost all men (about 95%) have a height within 6 inches of the mean (63-75 inches) – two standard deviations. If the standard deviation were zero, then all men would share an identical height of 69 inches. Three standard deviations account for 99.73% of the sample population being studied, assuming the distribution is normal or bell-shaped (see the 68-95-99.7 rule, or the empirical rule, for more information).

Let μ be the expected value (the average) of a random variable X with probability density function f: \[\mu \equiv \operatorname{\mathbb E}[X] = \int_{-\infty}^{+\infty} x \, f(x) \, {\mathrm d} x.\] The standard deviation σ of X is defined as \[\sigma \equiv \sqrt{\operatorname{\mathbb E}\left[ \left(X - \mu\right)^2 \right]} = \sqrt{ \int_{-\infty}^{+\infty} \left( x - \mu \right)^2 f(x) \ {\mathrm d} x \;}\ ,\] which can be shown to equal \(\sqrt{ \operatorname{\mathbb E}\left[X^2\right] - \left( \operatorname{\mathbb E}\left[ X \right] \right)^2 } \,.\)

In other words, the standard deviation is the square root of the variance of X.

The standard deviation of a probability distribution is the same as that of a random variable having that distribution.

Not all random variables have a standard deviation. If the distribution has fat tails going out to infinity, the standard deviation might not exist, because the integral might not converge. The normal distribution has tails going out to infinity, but its mean and standard deviation do exist, because the tails diminish quickly enough. The Pareto distribution with parameter \(\alpha \in (1,2]\) has a mean, but not a standard deviation (loosely speaking, the standard deviation is infinite). The Cauchy distribution has neither a mean nor a standard deviation.

In the case where X takes random values from a finite data set x₁, x₂, ..., x_N, with each value having the same probability, the standard deviation is

\[\sigma = \sqrt{\frac{1}{N}\ \left[ \left( x_1 - \mu \right)^2 + \left( x_2 - \mu \right)^2 + \cdots + \left( x_N - \mu \right)^2 \right] \;}\ , ~~\text{ where }~~ \mu \equiv \frac{1}{N} \left(x_1 + \cdots + x_N \right)\ ,\] Note: The above expression has a built-in bias. See the discussion on Bessel's correction further down below.

or, by using summation notation,

\[\sigma = \sqrt{ \frac{1}{N} \sum_{i=1}^N \left(x_i - \mu \right)^2 \;}\ , ~~\text{ where }~~ \mu \equiv \frac{1}{N} \sum_{i=1}^N x_i ~.\]

If, instead of having equal probabilities, the values have different probabilities, let x₁ have probability p₁, x₂ have probability p₂, ..., x_N have probability p_N . In this case, the standard deviation will be \[\sigma = \sqrt{ \sum_{i=1}^N p_i\left(x_i - \mu\right)^2 \;}\ , ~~\text{ where }~~ \mu \equiv \sum_{i=1}^N p_i x_i \,.\]

The standard deviation of a continuous real-valued random variable X with probability density function p(x) is \[\sigma = \sqrt{ \int_\mathbf{X} \left( x - \mu \right)^2\, p(x)\, {\mathrm d} x }\, , ~~\text{ where }~~ \mu \equiv \int_\mathbf{X} x\, p(x)\, {\mathrm d} x\, ,\]

and where the integrals are definite integrals taken for x ranging over X, which represents the set of possible values of the random variable X.

In the case of a parametric family of distributions, the standard deviation can often be expressed in terms of the parameters for the underlying distribution. For example, in the case of the log-normal distribution with parameters μ and σ for the underlying normal distribution, the standard deviation of the log-normal variable is given by the expression \[\sqrt{ \left(e^{\sigma^2} - 1\right)\ e^{2\mu + \sigma^2}} \,.\]

One can find the standard deviation of an entire population in cases (such as standardized testing) where every member of a population is sampled. In cases where that cannot be done, the standard deviation σ is estimated by examining a random sample taken from the population and computing a statistic of the sample, which is used as an estimate of the population standard deviation. Such a statistic is called an estimator, and the estimator (or the value of the estimator, namely the estimate) is called a sample standard deviation, and is denoted by s (possibly with modifiers).

Unlike in the case of estimating the population mean of a normal distribution, for which the sample mean is a simple estimator with many desirable properties (unbiased, efficient, maximum likelihood), there is no single estimator for the standard deviation with all these properties, and unbiased estimation of standard deviation is a very technically involved problem. Most often, the standard deviation is estimated using the corrected sample standard deviation (using N − 1), defined below, and this is often referred to as the "sample standard deviation", without qualifiers. However, other estimators are better in other respects: the uncorrected estimator (using N) yields lower mean squared error, while using N − 1.5 (for the normal distribution) almost eliminates bias.

The formula for the population standard deviation (of a finite population) can be applied to the sample, using the size of the sample as the size of the population (though the actual population size from which the sample is drawn may be much larger). This estimator, denoted by s_N, is known as the uncorrected sample standard deviation, or sometimes the standard deviation of the sample (considered as the entire population), and is defined as follows: \[s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N \left(x_i - \bar{x}\right)^2}\] where {x₁, x₂, …, x_N} are the observed values of the sample items and x̄ is the mean value of these observations, while the denominator N stands for the size of the sample: this is the square root of the sample variance, which is the average of the squared deviations about the sample mean.

This is a consistent estimator (it converges in probability to the population value as the number of samples goes to infinity), and is the maximum-likelihood estimate when the population is normally distributed. However, this is a biased estimator, as the estimates are generally too low. The bias decreases as sample size grows, dropping off as 1⁄N, and thus is most significant for small or moderate sample sizes; for N > 75, the bias is below 1%. Thus for very large sample sizes, the uncorrected sample standard deviation is generally acceptable. This estimator also has a uniformly smaller mean squared error than the corrected sample standard deviation.

If the biased sample variance (the second central moment of the sample, which is a downward-biased estimate of the population variance) is used to compute an estimate of the population's standard deviation, the result is \[s_N = \sqrt{\frac{1}{N} \sum_{i=1}^N \left(x_i - \bar{x}\right)^2}.\]

Here taking the square root introduces further downward bias, by Jensen's inequality, due to the square root's being a concave function. The bias in the variance is easily corrected, but the bias from the square root is more difficult to correct, and depends on the distribution in question.

An unbiased estimator for the variance is given by applying Bessel's correction, using N − 1 instead of N to yield the unbiased sample variance, denoted s: \[s^2 = \frac{1}{N - 1} \sum_{i=1}^N \left(x_i - \bar{x}\right)^2.\]

This estimator is unbiased if the variance exists and the sample values are drawn independently with replacement. N − 1 corresponds to the number of degrees of freedom in the vector of deviations from the mean, \(\textstyle(x_1 - \bar{x},\; \dots,\; x_n - \bar{x}).\)

Taking square roots reintroduces bias (because the square root is a nonlinear function which does not commute with the expectation, i.e. often \(E[\sqrt{X}]\neq \sqrt{E[X]}\)), yielding the corrected sample standard deviation, denoted by s: \[s = \sqrt{\frac{1}{N-1} \sum_{i=1}^N \left(x_i - \bar{x}\right)^2}.\]

As explained above, while s is an unbiased estimator for the population variance, s is still a biased estimator for the population standard deviation, though markedly less biased than the uncorrected sample standard deviation. This estimator is commonly used and generally known simply as the "sample standard deviation". The bias may still be large for small samples (N less than 10). As sample size increases, the amount of bias decreases. We obtain more information and the difference between \(\frac{1}{N}\) and \(\frac{1}{N-1}\) becomes smaller.

For unbiased estimation of standard deviation, there is no formula that works across all distributions, unlike for mean and variance. Instead, s is used as a basis, and is scaled by a correction factor to produce an unbiased estimate. For the normal distribution, an unbiased estimator is given by ⁠s/c₄⁠, where the correction factor (which depends on N) is given in terms of the gamma function, and equals: \[c_4(N)\,=\,\sqrt{\frac{2}{N-1}}\,\,\,\frac{\Gamma{\left(\frac{N}{2}\right)}}{\Gamma{\left(\frac{N-1}{2}\right)}}.\]

This arises because the sampling distribution of the sample standard deviation follows a (scaled) chi distribution, and the correction factor is the mean of the chi distribution.

An approximation can be given by replacing N − 1 with N − 1.5, yielding: \[\hat\sigma = \sqrt{\frac{1}{N - 1.5} \sum_{i=1}^N \left(x_i - \bar{x}\right)^2},\]

The error in this approximation decays quadratically (as ⁠1/N⁠), and it is suited for all but the smallest samples or highest precision: for N = 3 the bias is equal to 13%, and for N = 9 the bias is 3%.

A more accurate approximation is to replace N − 1.5 above with N − 1.5 + ⁠1/8(N − 1)⁠.

For other distributions, the correct formula depends on the distribution, but a rule of thumb is to use the further refinement of the approximation: \[\hat\sigma = \sqrt{\frac{1}{N - 1.5 - \frac{1}{4}\gamma_2} \sum_{i=1}^N \left(x_i - \bar{x}\right)^2},\]

where γ₂ denotes the population excess kurtosis. The excess kurtosis may be either known beforehand for certain distributions, or estimated from the data.

The standard deviation we obtain by sampling a distribution is itself not absolutely accurate, both for mathematical reasons (explained here by the confidence interval) and for practical reasons of measurement (measurement error). The mathematical effect can be described by the confidence interval or CI.

To show how a larger sample will make the confidence interval narrower, consider the following examples: A small population of N = 2 has only one degree of freedom for estimating the standard deviation. The result is that a 95% CI of the SD runs from 0.45 × SD to 31.9 × SD; the factors here are as follows:

\[\Pr\left(q_\frac{\alpha}{2} < k \frac{s^2}{\sigma^2} < q_{1 - \frac{\alpha}{2}}\right) = 1 - \alpha,\]

where \(q_p\) is the p-th quantile of the chi-square distribution with k degrees of freedom, and 1 − α is the confidence level. This is equivalent to the following:

\[\Pr\left(k\frac{s^2}{q_{1 - \frac{\alpha}{2}}} < \sigma^2 < k\frac{s^2}{q_{\frac{\alpha}{2}}}\right) = 1 - \alpha.\]

With k = 1, q_0.025 = 0.000982 and q_0.975 = 5.024. The reciprocals of the square roots of these two numbers give us the factors 0.45 and 31.9 given above.

A larger population of N = 10 has 9 degrees of freedom for estimating the standard deviation. The same computations as above give us in this case a 95% CI running from 0.69 × SD to 1.83 × SD. So even with a sample population of 10, the actual SD can still be almost a factor 2 higher than the sampled SD. For a sample population N = 100, this is down to 0.88 × SD to 1.16 × SD. To be more certain that the sampled SD is close to the actual SD we need to sample a large number of points.

Condensed: the full section is in Wikipedia.

For a set of N > 4 data spanning a range of values R, an upper bound on the standard deviation s is given by s = 0.6R. An estimate of the standard deviation for N > 100 data taken to be approximately normal follows from the heuristic that 95% of the area under the normal curve lies roughly two standard deviations to either side of the mean, so that, with 95% probability the total range of values R represents four standard deviations so that s ≈ R/4. This so-called range rule is useful in sample size estimation, as the range of possible values is easier to estimate than the standard deviation. Other divisors K(N) of the range such that s ≈ R/K(N) are available for other values of N and for non-normal distributions.

The standard deviation is invariant under changes in location, and scales directly with the scale of the random variable. Thus, for a constant c and random variables X and Y: \[\begin{align} \sigma(c) &= 0 \\ \sigma(X + c) &= \sigma(X), \\ \sigma(cX) &= |c| \sigma(X). \end{align}\]

The standard deviation of the sum of two random variables can be related to their individual standard deviations and the covariance between them:

\[\sigma(X + Y) = \sqrt{\operatorname{var}(X) + \operatorname{var}(Y) + 2 \,\operatorname{cov}(X,Y)}. \,\]

where \(\textstyle\operatorname{var} \,=\, \sigma^2\) and \(\textstyle\operatorname{cov}\) stand for variance and covariance, respectively.

The calculation of the sum of squared deviations can be related to moments calculated directly from the data. In the following formula, the letter E is interpreted to mean expected value, i.e., mean.

\[\sigma(X) = \sqrt{\operatorname E\left[\left(X - \operatorname E[X]\right)^2\right]} = \sqrt{\operatorname E\left[X^2\right] - \left(\operatorname E[X]\right)^2}.\]

The sample standard deviation can be computed as: \[s(X) = \sqrt{\frac{N}{N-1}} \sqrt{\operatorname E\left[\left(X - \operatorname E[X]\right)^2\right]}.\]

Condensed: the full section is in Wikipedia.