Bayesian inference

Bayesian inference (/ˈbeɪziən/ BAY-zee-ən or /ˈbeɪʒən/ BAY-zhən) is a method of statistical inference in which Bayes' theorem is used to calculate a probability of a hypothesis, given prior evidence, and update it as more information becomes available. Fundamentally, Bayesian inference uses a prior distribution to estimate posterior probabilities. Bayesian inference is an important technique in statistics, and especially in mathematical statistics. Bayesian updating is particularly important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a wide range of activities, including science, engineering, philosophy, medicine, sport, psychology, and law. In the philosophy of decision theory, Bayesian inference is closely related to subjective probability, often called "Bayesian probability".

Bayesian inference derives the posterior probability as a consequence of two antecedents: a prior probability and a "likelihood function" derived from a statistical model for the observed data. Bayesian inference computes the posterior probability according to Bayes' theorem:

\[P(H \mid E) = \frac{P(E \mid H) \cdot P(H)}{P(E)},\]

where

• \(H\) stands for any hypothesis whose probability may be affected by data (called evidence below). Often there are competing hypotheses, and the task is to determine which is the most probable.
• \(P(H)\), the prior probability, is the estimate of the probability of the hypothesis \(H\) before the data \(E\), the current evidence, is observed.
• \(E\), the evidence, corresponds to new data that were not used in computing the prior probability.
• \(P(H \mid E)\), the posterior probability, is the probability of \(H\) given \(E\), i.e., after \(E\) is observed. This is what we want to know: the probability of a hypothesis given the observed evidence.
• \(P(E \mid H)\) is the probability of observing \(E\) given \(H\) and is called the likelihood. As a function of \(E\) with \(H\) fixed, it indicates the compatibility of the evidence with the given hypothesis. The likelihood function is a function of the evidence, \(E\), while the posterior probability is a function of the hypothesis, \(H\).
• \(P(E)\) is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered (as is evident from the fact that the hypothesis \(H\) does not appear anywhere in the symbol, unlike for all the other factors) and hence does not factor into determining the relative probabilities of different hypotheses.
• \(P(E)>0\) (Else one has \(0/0\).)

For different values of \(H\), only the factors \(P(H)\) and \(P(E \mid H)\), both in the numerator, affect the value of \(P(H \mid E)\) – the posterior probability of a hypothesis is proportional to its prior probability (its inherent likeliness) and the newly acquired likelihood (its compatibility with the new observed evidence).

In cases where \(\neg H\) ("not \(H\)"), the logical negation of \(H\), is a valid likelihood, Bayes' rule can be rewritten as follows:

\[\begin{align} P(H \mid E) &= \frac{P(E \mid H) P(H)}{P(E)} \\ \\ &= \frac{P(E \mid H) P(H)}{P(E \mid H) P(H) + P(E \mid \neg H) P(\neg H)} \\ \\ &= \frac{1}{1 + \left(\frac{1}{P(H)} - 1\right) \frac{P(E \mid \neg H)}{P(E \mid H)} } \\ \end{align}\]

because

Condensed: the full section is in Wikipedia.

Bayesian updating is widely used and computationally convenient. However, it is not the only updating rule that might be considered rational.

Ian Hacking noted that traditional "Dutch book" arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. Hacking wrote: "And neither the Dutch book argument nor any other in the personalist arsenal of proofs of the probability axioms entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."

Indeed, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics") following the publication of Richard C. Jeffrey's rule, which applies Bayes' rule to the case where the evidence itself is assigned a probability. The additional hypotheses needed to uniquely require Bayesian updating have been deemed to be substantial, complicated, and unsatisfactory.

If evidence is simultaneously used to update belief over a set of exclusive and exhaustive propositions, Bayesian inference may be thought of as acting on this belief distribution as a whole.

Suppose a process is generating independent and identically distributed events \(E_n,\ n = 1, 2, 3, \ldots\), but the probability distribution is unknown. Let the event space \(\Omega\) represent the current state of belief for this process. Each model is represented by event \(M_m\). The conditional probabilities \(P(E_n \mid M_m)\) are specified to define the models. \(P(M_m)\) is the degree of belief in \(M_m\). Before the first inference step, \(\{P(M_m)\}\) is a set of initial prior probabilities. These must sum to 1, but are otherwise arbitrary.

Suppose that the process is observed to generate \(E \in \{E_n\}\). For each \(M \in \{M_m\}\), the prior \(P(M)\) is updated to the posterior \(P(M \mid E)\). From Bayes' theorem:

\[P(M \mid E) = \frac{P(E \mid M)}{\sum_m {P(E \mid M_m) P(M_m)}} \cdot P(M).\]

Upon observation of further evidence, this procedure may be repeated.

For a sequence of independent and identically distributed observations \(\mathbf{E} = (e_1, \dots, e_n)\), it can be shown by induction that repeated application of the above is equivalent to \[P(M \mid \mathbf{E}) = \frac{P(\mathbf{E} \mid M)}{\sum_m {P(\mathbf{E} \mid M_m) P(M_m)}} \cdot P(M),\] where \[P(\mathbf{E} \mid M) = \prod_k{P(e_k \mid M)}.\]

By parameterizing the space of models, the belief in all models may be updated in a single step. The distribution of belief over the model space may then be thought of as a distribution of belief over the parameter space. The distributions in this section are expressed as continuous, represented by probability densities, as this is the usual situation. The technique is, however, equally applicable to discrete distributions.

Let the vector \(\boldsymbol{\theta}\) span the parameter space. Let the initial prior distribution over \(\boldsymbol{\theta}\) be \(p(\boldsymbol{\theta} \mid \boldsymbol{\alpha})\), where \(\boldsymbol{\alpha}\) is a set of parameters to the prior itself, or hyperparameters. Let \(\mathbf{E} = (e_1, \dots, e_n)\) be a sequence of independent and identically distributed event observations, where all \(e_i\) are distributed as \(p(e \mid \boldsymbol{\theta})\) for some \(\boldsymbol{\theta}\). Bayes' theorem is applied to find the posterior distribution over \(\boldsymbol{\theta}\):

\[\begin{align} p(\boldsymbol{\theta} \mid \mathbf{E}, \boldsymbol{\alpha}) &= \frac{p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha})}{p(\mathbf{E} \mid \boldsymbol{\alpha})} \cdot p(\boldsymbol{\theta} \mid \boldsymbol{\alpha}) \\ &= \frac{p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha})}{\int p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha}) p(\boldsymbol{\theta} \mid \boldsymbol{\alpha}) \, d\boldsymbol{\theta}} \cdot p(\boldsymbol{\theta} \mid \boldsymbol{\alpha}), \end{align}\] where \[p(\mathbf{E} \mid \boldsymbol{\theta}, \boldsymbol{\alpha}) = \prod_k p(e_k \mid \boldsymbol{\theta}).\]

• \(x\), a data point in general. This may be a vector of values.
• \(\theta\), the parameter of the data point's distribution, i.e., \(x \sim p(x \mid \theta)\). This may be a vector of parameters.
• \(\alpha\), the hyperparameter of the parameter distribution, i.e., \(\theta \sim p(\theta \mid \alpha)\). This may be a vector of hyperparameters.
• \(\mathbf{X}\) is the sample, a set of \(n\) observed data points, i.e., \(x_1, \ldots, x_n\).
• \(\tilde{x}\), a new data point whose distribution is to be predicted.

• The prior distribution is the distribution of the parameter(s) before any data is observed, i.e. \(p(\theta \mid \alpha)\) . The prior distribution might not be easily determined; in such a case, one possibility may be to use the Jeffreys prior to obtain a prior distribution before updating it with newer observations.
• The sampling distribution is the distribution of the observed data conditional on its parameters, i.e. \(p(\mathbf{X} \mid \theta)\). This is also termed the likelihood, especially when viewed as a function of the parameter(s), sometimes written \(\operatorname{L}(\theta \mid \mathbf{X}) = p(\mathbf{X} \mid \theta)\).
• The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data marginalized over the parameter(s), i.e. \[p(\mathbf{X} \mid \alpha) = \int p(\mathbf{X} \mid \theta) p(\theta \mid \alpha) d\theta.\] It quantifies the agreement between data and expert opinion, in a geometric sense that can be made precise. If the marginal likelihood is 0 then there is no agreement between the data and expert opinion and Bayes' rule cannot be applied.
• The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is determined by Bayes' rule, which forms the heart of Bayesian inference:

\(p(\theta \mid \mathbf{X},\alpha) = \frac{p(\theta,\mathbf{X},\alpha)}{p(\mathbf{X},\alpha)} = \frac{p(\mathbf{X}\mid\theta,\alpha)p(\theta,\alpha)}{p(\mathbf{X}\mid\alpha)p(\alpha)}\) \(= \frac{p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha)}{p(\mathbf{X} \mid \alpha)} \propto p(\mathbf{X} \mid \theta,\alpha) p(\theta \mid \alpha).\)

This is expressed in words as "posterior is proportional to likelihood times prior", or sometimes as "posterior = likelihood times prior, over evidence".

• In practice, for almost all complex Bayesian models used in machine learning, the posterior distribution \(p(\theta \mid \mathbf{X},\alpha)\) is not obtained in a closed form distribution, mainly because the parameter space for \(\theta\) can be very high, or the Bayesian model retains certain hierarchical structure formulated from the observations \(\mathbf{X}\) and parameter \(\theta\). In such situations, we need to resort to approximation techniques.
• General case: Let \(P_Y^x\) be the conditional distribution of \(Y\) given \(X = x\) and let \(P_X\) be the distribution of \(X\). The joint distribution is then \(P_{X,Y} (dx,dy) = P_Y^x (dy) P_X (dx)\). The conditional distribution \(P_X^y\) of \(X\) given \(Y=y\) is then determined by

Condensed: the full section is in Wikipedia.

• The posterior predictive distribution is the distribution of a new data point, marginalized over the posterior: \[p(\tilde{x} \mid \mathbf{X},\alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \mathbf{X},\alpha) d\theta\]
• The prior predictive distribution is the distribution of a new data point, marginalized over the prior: \[p(\tilde{x} \mid \alpha) = \int p(\tilde{x} \mid \theta) p(\theta \mid \alpha) d\theta\]

Bayesian theory calls for the use of the posterior predictive distribution to do predictive inference, i.e., to predict the distribution of a new, unobserved data point. That is, instead of a fixed point as a prediction, a distribution over possible points is returned. Only this way is the entire posterior distribution of the parameter(s) used. By comparison, prediction in frequentist statistics often involves finding an optimum point estimate of the parameter(s), e.g., by maximum likelihood or maximum a posteriori estimation (MAP), and then plugging this estimate into the formula for the distribution of a data point. This has the disadvantage that it does not account for any uncertainty in the value of the parameter, and hence will underestimate the variance of the predictive distribution.

In some instances, frequentist statistics can work around this problem. For example, confidence intervals and prediction intervals in frequentist statistics when constructed from a normal distribution with unknown mean and variance are constructed using a Student's t-distribution. This correctly estimates the variance, due to the facts that (1) the average of normally distributed random variables is also normally distributed, and (2) the predictive distribution of a normally distributed data point with unknown mean and variance, using conjugate or uninformative priors, has a Student's t-distribution. In Bayesian statistics, however, the posterior predictive distribution can always be determined exactly, or at least to an arbitrary level of precision when numerical methods are used.

Both types of predictive distributions have the form of a compound probability distribution (as does the marginal likelihood). In fact, if the prior distribution is a conjugate prior, such that the prior and posterior distributions come from the same family, it can be seen that both prior and posterior predictive distributions also come from the same family of compound distributions. The only difference is that the posterior predictive distribution uses the updated values of the hyperparameters (applying the Bayesian update rules given in the conjugate prior article), while the prior predictive distribution uses the values of the hyperparameters that appear in the prior distribution.

\(\frac{P(E \mid M)}{P(E)} > 1 \Rightarrow P(E \mid M) > P(E)\). That is, if the model were true, the evidence would be more likely than is predicted by the current state of belief. The reverse applies for a decrease in belief. If the belief does not change, \(\frac{P(E \mid M)}{P(E)} = 1 \Rightarrow P(E \mid M) = P(E)\). That is, the evidence is independent of the model. If the model were true, the evidence would be exactly as likely as predicted by the current state of belief.

If \(P(M) = 0\) then \(P(M \mid E) = 0\). If \(P(M) = 1\) and \(P(E) > 0\), then \(P(M|E) = 1\). This can be interpreted to mean that hard convictions are insensitive to counter-evidence.

The former follows directly from Bayes' theorem. The latter can be derived by applying the first rule to the event "not \(M\)" in place of "\(M\)", yielding "if \(1 - P(M) = 0\), then \(1 - P(M \mid E) = 0\)", from which the result immediately follows.

Consider the behaviour of a belief distribution as it is updated a large number of times with independent and identically distributed trials. For sufficiently nice prior probabilities, the Bernstein-von Mises theorem gives that in the limit of infinite trials, the posterior converges to a Gaussian distribution independent of the initial prior under some conditions firstly outlined and rigorously proven by Joseph L. Doob in 1948, namely if the random variable in consideration has a finite probability space. The more general results were obtained later by the statistician David A. Freedman who published in two seminal research papers in 1963 and 1965 when and under what circumstances the asymptotic behaviour of posterior is guaranteed. His 1963 paper treats, like Doob (1949), the finite case and comes to a satisfactory conclusion. However, if the random variable has an infinite but countable probability space (i.e., corresponding to a die with infinite many faces) the 1965 paper demonstrates that for a dense subset of priors the Bernstein-von Mises theorem is not applicable. In this case there is almost surely no asymptotic convergence. Later in the 1980s and 1990s Freedman and Persi Diaconis continued to work on the case of infinite countable probability spaces. To summarise, there may be insufficient trials to suppress the effects of the initial choice, and especially for large (but finite) systems the convergence might be very slow.

In parameterized form, the prior distribution is often assumed to come from a family of distributions called conjugate priors. The usefulness of a conjugate prior is that the corresponding posterior distribution will be in the same family, and the calculation may be expressed in closed form.