Chi-squared distribution

In probability theory and statistics, the \(\chi^2\)-distribution with \(k\) degrees of freedom is the distribution of a sum of the squares of \(k\) independent standard normal random variables.

The chi-squared distribution \(\chi^2_k\) is a special case of the gamma distribution and the univariate Wishart distribution. Specifically if \(X \sim \chi^2_k\) then \(X \sim \text{Gamma}(\alpha=\frac{k}{2}, \theta=2)\) (where \(\alpha\) is the shape parameter and \(\theta\) the scale parameter of the gamma distribution) and \(X \sim \text{W}_1(1,k)\).

The scaled chi-squared distribution \(s^2 \chi^2_k\) is a reparametrization of the gamma distribution and the univariate Wishart distribution. Specifically if \(X \sim s^2 \chi^2_k\) then \(X \sim \text{Gamma}(\alpha=\frac{k}{2}, \theta=2 s^2)\) and \(X \sim \text{W}_1(s^2,k)\).

The chi-squared distribution is one of the most widely used probability distributions in inferential statistics, notably in hypothesis testing and in construction of confidence intervals. This distribution is sometimes called the central chi-squared distribution, a special case of the more general noncentral chi-squared distribution.

The chi-squared distribution is used in the common chi-squared tests for goodness of fit of an observed distribution to a theoretical one, the independence of two criteria of classification of qualitative data, and in finding the confidence interval for estimating the population standard deviation of a normal distribution from a sample standard deviation. Many other statistical tests also use this distribution, such as Friedman's analysis of variance by ranks.

If Z₁, ..., Z_k are independent, standard normal random variables, then the sum of their squares, \[X\ = \sum_{i=1}^k Z_i^2,\] is distributed according to the chi-squared distribution with k degrees of freedom.

This is usually denoted as \[X\ \sim\ \chi^2(k)\ \ \text{or}\ \ X\ \sim\ \chi^2_k.\]

The chi-squared distribution has one parameter: a positive integer k that specifies the number of degrees of freedom (the number of random variables Z_i being summed).

The chi-squared distribution is used primarily in hypothesis testing, and to a lesser extent for confidence intervals for population variance when the underlying distribution is normal. Unlike more widely known distributions such as the normal distribution and the exponential distribution, the chi-squared distribution is not as often applied in the direct modeling of natural phenomena. It arises in the following hypothesis tests, among others:

• Chi-squared test of independence in contingency tables
• Chi-squared test of goodness of fit of observed data to hypothetical distributions
• Likelihood-ratio test for nested models
• Log-rank test in survival analysis
• Cochran-Mantel-Haenszel test for stratified contingency tables
• Wald test
• Score test

It is also a component of the definition of the t-distribution and the F-distribution used in t-tests, analysis of variance, and regression analysis.

The primary reason for which the chi-squared distribution is extensively used in hypothesis testing is its relationship to the normal distribution. Many hypothesis tests use a test statistic, such as the t-statistic in a t-test. For these hypothesis tests, as the sample size, n, increases, the sampling distribution of the test statistic approaches the normal distribution (central limit theorem). Because the test statistic (such as t) is asymptotically normally distributed, provided the sample size is sufficiently large, the distribution used for hypothesis testing may be approximated by a normal distribution. Testing hypotheses using a normal distribution is well understood and relatively easy. The simplest chi-squared distribution is the square of a standard normal distribution. So wherever a normal distribution could be used for a hypothesis test, a chi-squared distribution could be used.

Suppose that \(Z\) is a random variable sampled from the standard normal distribution, where the mean is \(0\) and the variance is \(1\): \(Z \sim N(0,1)\). Now, consider the random variable \(X = Z^2\). The distribution of the random variable \(X\) is an example of a chi-squared distribution: \(\ X\ \sim\ \chi^2_1\). The subscript 1 indicates that this particular chi-squared distribution is constructed from only 1 standard normal distribution. A chi-squared distribution constructed by squaring a single standard normal distribution is said to have 1 degree of freedom. Thus, as the sample size for a hypothesis test increases, the distribution of the test statistic approaches a normal distribution. Just as extreme values of the normal distribution have low probability (and give small p-values), extreme values of the chi-squared distribution have low probability.

Lancaster shows the connections among the binomial, normal, and chi-squared distributions, as follows. De Moivre and Laplace established that a binomial distribution could be approximated by a normal distribution. Specifically they showed the asymptotic normality of the random variable

\[\chi = \frac{m - Np}{\sqrt{Npq}}\]

Condensed: the full section is in Wikipedia.

The probability density function (pdf) of the chi-squared distribution is \[f(x;\,k) = \begin{cases} \dfrac{x^{k/2 -1} e^{-x/2}}{2^{k/2} \, \Gamma{\left(\frac k 2 \right)}}, & x > 0; \\ 0, & \text{otherwise}. \end{cases}\] where \(\Gamma(k/2)\) denotes the gamma function, which has closed-form values for integer \(k\).

For derivations of the pdf in the cases of one, two and \(k\) degrees of freedom, see Proofs related to chi-squared distribution.

Its cumulative distribution function is: \[F(x;\,k) = \frac{\gamma{\left(\frac{k}{2},\,\frac{x}{2}\right)}}{\Gamma{\left(\frac{k}{2}\right)}} = P{\left(\frac{k}{2},\,\frac{x}{2}\right)},\] where \(\gamma(s,t)\) is the lower incomplete gamma function and \(P(s,t)\) is the regularized gamma function.

In a special case of \(k = 2\) this function has the simple form: \[F(x;\,2) = 1 - e^{-x/2}\] which can be easily derived by integrating \(f(x;\,2) = \frac{1}{2}e^{-x/2}\) directly. The integer recurrence of the gamma function makes it easy to compute \(F(x;\,k)\) for other small, even \(k\).

Tables of the chi-squared cumulative distribution function are widely available and the function is included in many spreadsheets and all statistical packages.

Letting \(z \equiv x/k\), Chernoff bounds on the lower and upper tails of the CDF may be obtained. For the cases when 0 < z < 1\) (which include all of the cases when this CDF is less than half): \(F(z k;\,k) \leq (z e^{1-z})^{k/2}.\)

The tail bound for the cases when \(z > 1\), similarly, is \[1-F(z k;\,k) \leq (z e^{1-z})^{k/2}.\]

For another approximation for the CDF modeled after the cube of a Gaussian, see under Noncentral chi-squared distribution.

The following is a special case of Cochran's theorem.

Theorem. If \(Z_1,...,Z_n\) are independent identically distributed (i.i.d.), standard normal random variables, then \(\sum_{t=1}^n \left(Z_t - \bar Z\right)^2 \sim \chi^2_{n-1}\) where \(\bar Z = \frac{1}{n} \sum_{t=1}^n Z_t.\)

Proof. Let \(Z\sim\mathcal{N}(\bar 0,1\!\!1)\) be a vector of \(n\) independent normally distributed random variables, and \(\bar Z\) their average. Then \(\sum_{t=1}^n(Z_t-\bar Z)^2 ~=~ \sum_{t=1}^n Z_t^2 -n\bar Z^2 ~=~ Z^\top[1\!\!1 -{\textstyle\frac1n}\bar 1\bar 1^\top]Z ~=:~ Z^\top\!M Z\) where \(1\!\!1\) is the identity matrix and \(\bar 1\) the all ones vector. \(M\) has one eigenvector \(b_1:={\textstyle\frac{1}{\sqrt{n}}} \bar 1\) with eigenvalue \(0\), and \(n-1\) eigenvectors \(b_2,...,b_n\) (all orthogonal to \(b_1\)) with eigenvalue \(1\), which can be chosen so that \(Q:=(b_1,...,b_n)\) is an orthogonal matrix. Since also \(X:=Q^\top\!Z\sim\mathcal{N}(\bar 0,Q^\top\!1\!\!1 Q) =\mathcal{N}(\bar 0,1\!\!1)\), we have \(\sum_{t=1}^n(Z_t-\bar Z)^2 ~=~ Z^\top\!M Z ~=~ X^\top\!Q^\top\!M Q X ~=~ X_2^2+...+X_n^2 ~\sim~ \chi^2_{n-1},\) which proves the claim.

It follows from the definition of the chi-squared distribution that the sum of independent chi-squared variables is also chi-squared distributed. Specifically, if \(X_i,i=\overline{1,n}\) are independent chi-squared variables with \(k_i\), \(i=\overline{1,n}\) degrees of freedom, respectively, then \(Y = X_1 + \cdots + X_n\) is chi-squared distributed with \(k_1 + \cdots + k_n\) degrees of freedom.

The sample mean of \(n\) i.i.d. chi-squared variables of degree \(k\) is distributed according to a gamma distribution with shape \(\alpha\) and scale \(\theta\) parameters: \[\overline X = \frac{1}{n} \sum_{i=1}^n X_i \sim \operatorname{Gamma}\left(\alpha{=}\tfrac{n k}{2}, \,\theta{=}\tfrac{2}{n} \right) \qquad \text{where } X_i \sim \chi^2(k)\]

Asymptotically, given that for a shape parameter \(\alpha\) going to infinity, a Gamma distribution converges towards a normal distribution with expectation \(\mu = \alpha \theta\) and variance \(\sigma^2 = \alpha \theta^2\), the sample mean converges towards:

\[\overline X \xrightarrow{n \to \infty} N{\left(\mu{=}k, \, \sigma^2{=}\tfrac{2k}{n} \right)}\]

Note that we would have obtained the same result invoking instead the central limit theorem, noting that for each chi-squared variable of degree \(k\) the expectation is \(k\), and its variance \(2k\) (and hence the variance of the sample mean \(\overline{X}\) being \(\sigma^2 = \tfrac{2k}{n}\)).

The differential entropy is given by \[\begin{aligned} h &= \int_0^\infty f(x;\,k) \ln f(x;\,k) \, dx \\ &= \frac k 2 + \ln \left[2\,\Gamma{\left(\frac k 2 \right)}\right] + \left(1-\frac k 2 \right) \psi\!\left(\frac k 2 \right), \end{aligned}\] where \(\psi(x)\) is the Digamma function.

The chi-squared distribution is the maximum entropy probability distribution for a random variate \(X\) for which \(\operatorname{E}(X)=k\) and \(\operatorname{E}(\ln(X))=\psi(k/2)+\ln(2)\) are fixed. Since the chi-squared is in the family of gamma distributions, this can be derived by substituting appropriate values in the Expectation of the log moment of gamma. For derivation from more basic principles, see the derivation in moment-generating function of the sufficient statistic.

The noncentral moments (raw moments) of a chi-squared distribution with \(k\) degrees of freedom are given by \[\begin{aligned} \operatorname{E}(X^m) &= k (k+2) (k+4) \cdots (k+2m-2) \\[1ex] &= 2^m \frac{\Gamma{\left(m+\frac{k}{2}\right)}}{\Gamma{\left(\frac{k}{2}\right)}}. \end{aligned}\]

The cumulants are readily obtained by a power series expansion of the logarithm of the characteristic function: \[\kappa_n = 2^{n-1}(n-1)!\,k\] with cumulant generating function \(\ln \operatorname{E}[e^{tX}] = - \frac{k}{2} \ln(1-2t)\).

The chi-squared distribution exhibits strong concentration around its mean. The standard Laurent-Massart bounds are: \[\Pr(X - k \ge 2 \sqrt{k x} + 2x) \le e^{-x}\] \[\Pr(k - X \ge 2 \sqrt{k x}) \le e^{-x}\] One consequence is that, if \(Z \sim N(0, 1)^k\) is a Gaussian random vector in \(\R^k\), then as the dimension \(k\) grows, the squared length of the vector is concentrated tightly around \(k\) with a width \(k^{1/2 + \alpha}\):\[\Pr\left(\left\|Z\right\|^2 \in \left[k - 2k^{1/2+\alpha}, \; k + 2k^{1/2+\alpha} + 2k^{\alpha}\right]\right) \geq 1-e^{-k^\alpha}\]where the exponent \(\alpha\) can be chosen as any value in \(\R\).

Since the cumulant generating function for \(\chi^2(k)\) is \(K(t) = -\frac k2 \ln(1-2t)\), and its convex dual is \(K^*(q) = \frac{1}{2} \left(q - k + k\ln\frac{k}{q}\right)\), the standard Chernoff bound yields\[\begin{aligned} \ln \Pr(X \geq (1 + \varepsilon) k) &\leq -\frac{k}{2} \left( \varepsilon - \ln(1+\varepsilon)\right) \\ \ln \Pr(X \leq (1 - \varepsilon) k) &\leq -\frac{k}{2} \left(-\varepsilon - \ln(1-\varepsilon)\right) \end{aligned}\]where \(0 < \varepsilon < 1\). By the union bound,\[Pr(X \in (1\pm \varepsilon ) k ) \geq 1 - 2 e^{-\frac k2 (\frac{1}{2} \varepsilon^2 - \frac 13 \varepsilon^3)}\]This result is used in proving the Johnson-Lindenstrauss lemma.

By the central limit theorem, because the chi-squared distribution is the sum of \(k\) independent random variables with finite mean and variance, it converges to a normal distribution for large \(k\). For many practical purposes, for \(k>50\) the distribution is sufficiently close to a normal distribution, so the difference is ignorable. Specifically, if \(X \sim \chi^2(k)\), then as \(k\) tends to infinity, the distribution of \((X-k)/\sqrt{2k}\) tends to a standard normal distribution. However, convergence is slow as the skewness is \(\sqrt{8/k}\) and the excess kurtosis is \(12/k\).

The sampling distribution of \(\ln(\chi^2)\) converges to normality much faster than the sampling distribution of \(\chi^2\), as the logarithmic transform removes much of the asymmetry.

Other functions of the chi-squared distribution converge more rapidly to a normal distribution. Some examples are:

• If \(X \sim \chi^2(k)\) then \(\sqrt{2X}\) is approximately normally distributed with mean \(\sqrt{2k-1}\) and unit variance (1922, by R. A. Fisher, see (18.23), p. 426 of Johnson).
• If \(X \sim \chi^2(k)\) then \(\sqrt[3]{X/k}\) is approximately normally distributed with mean \(1-\frac{2}{9k}\) and variance \(\frac{2}{9k} .\) This is known as the Wilson-Hilferty transformation, see (18.24), p. 426 of Johnson.
  • This normalizing transformation leads directly to the commonly used median approximation \(k\bigg(1-\frac{2}{9k}\bigg)^3\;\) by back-transforming from the mean, which is also the median, of the normal distribution.

• As \(k\to\infty\), \((\chi^2_k-k)/\sqrt{2k} ~ \xrightarrow{d}\ N(0,1) \,\) (normal distribution)
• \(\chi_k^2 \sim {\chi'}^2_k(0)\) (noncentral chi-squared distribution with non-centrality parameter \(\lambda = 0\))
• If \(Y \sim \mathrm{F}(\nu_1, \nu_2)\) then \(X = \lim_{\nu_2 \to \infty} \nu_1 Y\) has the chi-squared distribution \(\chi^2_{\nu_{1}}\)
  • As a special case, if \(Y \sim \mathrm{F}(1, \nu_2)\,\) then \(X = \lim_{\nu_2 \to \infty} Y\,\) has the chi-squared distribution \(\chi^2_{1}\)
• \(\left\|\boldsymbol{N}_{i=1,\ldots,k} (0,1) \right\|^2 \sim \chi^2_k\) (The squared norm of k standard normally distributed variables is a chi-squared distribution with k degrees of freedom)
• If \(X \sim \chi^2_\nu\,\) and \(c>0 \,\), then \(cX \sim \Gamma(k=\nu/2, \theta=2c)\,\). (gamma distribution)
• If \(X \sim \chi^2_k\) then \(\sqrt{X} \sim \chi_k\) (chi distribution)
• If \(X \sim \chi^2_2\), then \(X \sim \operatorname{exp}(1/2)\) is an exponential distribution. (See gamma distribution for more.)
• If \(X \sim \chi^2_{2k}\), then \(X \sim \operatorname{Erlang}(k, 1/2)\) is an Erlang distribution.
• If \(X \sim \operatorname{Erlang}(k,\lambda)\), then \(2\lambda X\sim \chi^2_{2k}\)
• If \(X \sim \operatorname{Rayleigh}(1)\,\) (Rayleigh distribution) then \(X^2 \sim \chi^2_2\,\)
• If \(X \sim \operatorname{Maxwell}(1)\,\) (Maxwell distribution) then \(X^2 \sim \chi^2_3\,\)
• If \(X \sim \chi^2_\nu\) then \(\tfrac{1}{X} \sim \operatorname{Inv-}\chi^2_\nu\,\) (Inverse-chi-squared distribution)
• The chi-squared distribution is a special case of type III Pearson distribution
• If \(X \sim \chi^2_{\nu_1}\,\) and \(Y \sim \chi^2_{\nu_2}\,\) are independent then \(\tfrac{X}{X+Y} \sim \operatorname{Beta}(\tfrac{\nu_1}{2}, \tfrac{\nu_2}{2})\,\) (beta distribution)
• If \(X \sim \operatorname{U}(0,1)\,\) (uniform distribution) then \(-2\log(X) \sim \chi^2_2\,\)
• If \(X_i \sim \operatorname{Laplace}(\mu,\beta)\,\) then \(\sum_{i=1}^n \frac{2 |X_i-\mu|}{\beta} \sim \chi^2_{2n}\,\)
• If \(X_i\) follows the generalized normal distribution (version 1) with parameters \(\mu,\alpha,\beta\) then \(\sum_{i=1}^n \frac{2 |X_i-\mu|^\beta}{\alpha} \sim \chi^2_{2n/\beta}\,\)
• The chi-squared distribution is a transformation of Pareto distribution
• Student's t-distribution is a transformation of chi-squared distribution
• Student's t-distribution can be obtained from chi-squared distribution and normal distribution
• The noncentral beta distribution can be obtained as a transformation of chi-squared distribution and noncentral chi-squared distribution
• The noncentral t-distribution can be obtained from normal distribution and chi-squared distribution

A chi-squared variable with \(k\) degrees of freedom is defined as the sum of the squares of \(k\) independent standard normal random variables.

If \(Y\) is a \(k\)-dimensional Gaussian random vector with mean vector \(\mu\) and rank \(k\) covariance matrix \(C\), then \(X = (Y-\mu )^\mathsf{T}C^{-1}(Y-\mu)\) is chi-squared distributed with \(k\) degrees of freedom.

The sum of squares of statistically independent unit-variance Gaussian variables which do not have mean zero yields a generalization of the chi-squared distribution called the noncentral chi-squared distribution.

\[\frac{1}{\tilde{w}^\mathsf{T} \Sigma \tilde{w}} \sim \chi_1^2.\]where \(\tilde{w} = (w_1/X_1, \dots, w_p/X_p)\).

Condensed: the full section is in Wikipedia.