Logistic function

A logistic function or logistic curve is a common S-shaped curve (sigmoid curve) with the equation

\[f(x) = \frac{L}{1 + e^{-k(x-x_0)}}\]

where

• \(L\) is the carrying capacity, the supremum of the values of the function;
• \(k\) is the logistic growth rate, the steepness of the curve; and
• \(x_0\) is the \(x\) value of the function's midpoint.

The logistic function has domain the real numbers, the limit as \(x\) tends to \(-\infty\) is 0, and the limit as \(x\) tends to \(+\infty\) is \(L\).

The exponential function with negated argument (\(e^{-x}\)) is used to define the standard logistic function where \(L=1, k=1, x_0=0\), which has the equation \[f(x) = \frac{1}{1 + e^{-x}}\] and is sometimes simply called the sigmoid function. It is also sometimes called the expit, being the inverse function of the logit.

The logistic function finds applications in a range of fields, including biology (especially ecology), biomathematics, chemistry, demography, economics, geoscience, mathematical psychology, probability, sociology, political science, linguistics, statistics, and artificial neural networks. There are various generalizations, depending on the field.

The logistic function was introduced in a series of three papers by Pierre François Verhulst between 1838 and 1847, who devised it as a model of population growth by adjusting the exponential growth model, under the guidance of Adolphe Quetelet. Verhulst first devised the function in the mid 1830s, publishing a brief note in 1838, then presented an expanded analysis and named the function in 1844 (published 1845); the third paper adjusted the correction term in his model of Belgian population growth.

The initial stage of growth is approximately exponential (geometric); then, as saturation begins, the growth slows to linear (arithmetic), and at maturity, growth approaches the limit with an exponentially decaying gap, like the initial stage in reverse.

Verhulst did not explain the choice of the term "logistic" (French: logistique), but it is presumably in contrast to the logarithmic curve, and by analogy with arithmetic and geometric. His growth model is preceded by a discussion of arithmetic growth and geometric growth (whose curve he calls a logarithmic curve, instead of the modern term exponential curve), and thus "logistic growth" is presumably named by analogy, logistic being from Ancient Greek: λογιστικός, romanized: logistikós, a traditional division of Greek mathematics.

As a word derived from ancient Greek mathematical terms, the name of this function is unrelated to the military and management term logistics, which is instead from French: logis "lodgings", though some believe the Greek term also influenced logistics; see Logistics § Origin for details.

The standard logistic function is the logistic function with parameters \(k = 1\), \(x_0 = 0\), \(L = 1\), which yields

\[f(x) = \frac{1}{1 + e^{-x}} = \frac{e^x}{e^x + 1} = \frac{e^{x/2}}{e^{x/2} + e^{-x/2}}.\]

In practice, due to the nature of the exponential function \(e^{-x}\), it is often sufficient to compute the standard logistic function for \(x\) over a small range of real numbers, such as a range contained in [−6, +6], as it quickly converges very close to its saturation values of 0 and 1.

The logistic function has the symmetry property that

\[1 - f(x) = f(-x).\]

This reflects that the growth from 0 when \(x\) is small is symmetric with the decay of the gap to the limit (1) when \(x\) is large.

Further, \(x \mapsto f(x) - 1/2\) is an odd function.

The sum of the logistic function and its reflection about the vertical axis, \(f(-x)\), is

\[\frac{1}{1 + e^{-x}} + \frac{1}{1 + e^{-(-x)}} = \frac{e^x}{e^x + 1} + \frac{1}{e^x + 1} = 1.\]

The logistic function is thus rotationally symmetrical about the point (0, 1/2).

The logistic function is the inverse of the natural logit function

\[\operatorname{logit} p = \log \frac p {1-p} \quad \text{ for }\, 0

and so converts the logarithm of odds into a probability.

\(\operatorname{logistic}(\operatorname{logit}(p)) = \dfrac{1}{1+e^{-\log\left(\frac{p}{1-p}\right)}} = \dfrac{1}{1+e^{\log\left(\left(\frac{p}{1-p}\right)^{-1}\right)}} = \dfrac{1}{1+\frac{1-p}{p}} = \dfrac{p}{p+1-p} = p\)

The conversion from the log-likelihood ratio of two alternatives also takes the form of a logistic curve.

The logistic function is an offset and scaled hyperbolic tangent function: \[f(x) = \frac12 + \frac12 \tanh\left(\frac{x}{2}\right),\] or \[\tanh(x) = 2 f(2x) - 1.\]

This follows from \[\begin{aligned} \tanh(x) & = \frac{e^x - e^{-x}}{e^x + e^{-x}} \\ &= \frac{e^x \cdot \left(1 - e^{-2x}\right)}{e^x \cdot \left(1 + e^{-2x}\right)} \\ &= f(2x) - \frac{e^{-2x}}{1 + e^{-2x}} \\ &= f(2x) - \frac{e^{-2x} + 1 - 1}{1 + e^{-2x}} \\ &= 2f(2x) - 1. \end{aligned}\]

The hyperbolic-tangent relationship leads to another form for the logistic function's derivative:

\[\frac{d}{dx} f(x) = \frac14 \operatorname{sech}^2\left(\frac{x}{2}\right),\]

which ties the logistic function into the logistic distribution.

Geometrically, the hyperbolic tangent function is the hyperbolic angle on the unit hyperbola \(x^2 - y^2 = 1\), which factors as \((x + y)(x - y) = 1\), and thus has asymptotes the lines through the origin with slope ⁠\(-1\)⁠ and with slope ⁠\(1\)⁠, and vertex at ⁠\((1, 0)\)⁠ corresponding to the range and midpoint (⁠\({{{1}}}\)⁠) of tanh. Analogously, the logistic function can be viewed as the hyperbolic angle on the hyperbola \(xy - y^2 = 1\), which factors as \(y(x - y) = 1\), and thus has asymptotes the lines through the origin with slope ⁠\(0\)⁠ and with slope ⁠\(1\)⁠, and vertex at ⁠\((2, 1)\)⁠, corresponding to the range and midpoint (⁠\(1/2\)⁠) of the logistic function.

Parametrically, hyperbolic cosine and hyperbolic sine give coordinates on the unit hyperbola: \(\left( (e^t + e^{-t})/2, (e^t - e^{-t})/2\right)\), with quotient the hyperbolic tangent. Similarly, \(\bigl(e^{t/2} + e^{-t/2}, e^{t/2}\bigr)\) parametrizes the hyperbola \(xy - y^2 = 1\), with quotient the logistic function. These correspond to linear transformations (and rescaling the parametrization) of the hyperbola \(xy = 1\), with parametrization \((e^{-t}, e^t)\): the parametrization of the hyperbola for the logistic function corresponds to \(t/2\) and the linear transformation \(\bigl( \begin{smallmatrix} 1 & 1\\ 0 & 1 \end{smallmatrix} \bigr)\), while the parametrization of the unit hyperbola (for the hyperbolic tangent) corresponds to the linear transformation \(\tfrac{1}{2}\bigl( \begin{smallmatrix} 1 & 1\\ -1 & 1 \end{smallmatrix} \bigr)\).

The standard logistic function has an easily calculated derivative. The derivative is known as the density of the logistic distribution:

\[f(x) = \frac{1}{1 + e^{-x}} = \frac{e^x}{1 + e^x},\]

\[\begin{aligned} \frac{d}{dx} f(x) &= \frac{e^x \cdot (1 + e^x) - e^x \cdot e^x}{{\left(1 + e^x\right)}^2} \\[1ex] &= \frac{e^x}{{\left(1 + e^x\right)}^2} \\[1ex] &= \frac{e^x}{1 + e^x} \cdot \frac{1}{1 + e^x} \\[1ex] &= \frac{e^x}{1 + e^x} \left(\frac{1 + e^x - e^x}{1 + e^x}\right) \\[1.2ex] &= \frac{e^x}{1 + e^x} \left(1-\frac{e^x}{1 + e^x}\right) \\[1.2ex] &= f(x)\left(1 - f(x)\right) \end{aligned}\]from which all higher derivatives can be derived algebraically. For example, \(f'' = (1-2f)(1-f)f\).

The logistic distribution is a location, scale family, which corresponds to parameters of the logistic function. If ⁠\(L = 1\)⁠ is fixed, then the midpoint ⁠\(x_0\)⁠ is the location and the slope ⁠\(k\)⁠ is the scale.

Conversely, its antiderivative can be computed by the substitution \(u = 1 + e^x\), since

\[f(x) = \frac{e^x}{1 + e^x} = \frac{u'}{u},\]

so (dropping the constant of integration)

\[\int \frac{e^x}{1 + e^x}\,dx = \int \frac{1}{u}\,du = \ln u = \ln (1 + e^x).\]

In artificial neural networks, this is known as the softplus function and (with scaling) is a smooth approximation of the ramp function, just as the logistic function (with scaling) is a smooth approximation of the Heaviside step function.

The standard logistic function is analytic on the whole real line since \(f : \mathbb{R} \to \mathbb{R}\), \(f(x) = \frac{1}{1+e^{-x}} = h(g(x))\) where \(g : \mathbb{R} \to \mathbb{R}\), \(g(x) = 1 + e^{-x}\) and \(h : (0, \infty) \to (0, \infty)\), \(h(x) = \frac{1}{x}\) are analytic on their domains, and the composition of analytic functions is again analytic.

A formula for the nth derivative of the standard logistic function is

\[\frac{d^n f}{dx^n} = \sum_{i=1}^n \frac{\left(\sum_{j=1}^n {\left(-1\right)}^{i+j} \binom{i}{j} j^n\right) e^{-ix}}{{\left(1+e^{-x}\right)}^{i+1}}\]

therefore its Taylor series about the point \(a\) is

\[f(x) = f(a) + \sum_{n=1}^{\infty} \sum_{i=1}^n \frac{\left(\sum_{j=1}^n {\left(-1\right)}^{i+j} \binom{i}{j} j^n\right) e^{-ia}}{{\left(1 + e^{-a}\right)}^{i+1}} \frac{{\left(x-a\right)}^n}{n!} .\]

The unique standard logistic function is the solution of the simple first-order non-linear ordinary differential equation

\[\frac{d}{dx}f(x) = f(x)\big(1 - f(x)\big)\]

with boundary condition \(f(0) = 1/2\). This equation is the continuous version of the logistic map. Note that the reciprocal logistic function is solution to a simple first-order linear ordinary differential equation.

The qualitative behavior is easily understood in terms of the phase line: the derivative is 0 when the function is 1; and the derivative is positive for \(f\) between 0 and 1, and negative for \(f\) above 1 or less than 0 (though negative populations do not generally accord with a physical model). This yields an unstable equilibrium at 0 and a stable equilibrium at 1, and thus for any function value greater than 0 and less than 1, it grows to 1.

The logistic equation is a special case of the Bernoulli differential equation and has the following solution:

\[f(x) = \frac{e^x}{e^x + C}.\]

Choosing the constant of integration \(C = 1\) gives the other well known form of the definition of the logistic curve:

Condensed: the full section is in Wikipedia.

When the capacity \(L = 1\), the value of the logistic function is in the range ⁠\((0, 1)\)⁠ and can be interpreted as a probability p. In more detail, p can be interpreted as the probability of one of two alternatives (the parameter of a Bernoulli distribution); the two alternatives are complementary, so the probability of the other alternative is \(q = 1 - p\) and \(p + q = 1\). The two alternatives are coded as 1 and 0, corresponding to the limiting values as \(x \to \pm \infty\).

In this interpretation the input x is the log-odds for the first alternative (relative to the second, measured in "logistic units" or logits), and so ⁠\(e^x\)⁠ is the odds for the first alternative (relative to the second). Given odds for an event of \(O = O:1\) (⁠\(O\)⁠ against 1), the probability is the ratio of "for" over "for plus against", \(O/(O+1)\). We see that the logistic function, \(e^x/(e^x + 1) = 1/(1 + e^{-x}) = p\), is the probability of the first alternative.

Conversely, x is the log-odds against the second alternative, ⁠\(-x\)⁠ is the log-odds for the second alternative, \(e^{-x}\) is the odds for the second alternative, and \(e^{-x}/(e^{-x} + 1) = 1/(1 + e^x) = q = 1 - p\) is the probability of the second alternative.

This can be framed more symmetrically in terms of two inputs, ⁠\(x_0\)⁠ and ⁠\(x_1\)⁠, which then generalizes naturally to more than two alternatives. Given two real number inputs, ⁠\(x_0\)⁠ and ⁠\(x_1\)⁠, interpreted as logits, their difference \(x_1 - x_0\) is the log-odds for option 1 (the log-odds against option 0), \(e^{x_1 - x_0}\) is the odds, \(e^{x_1 - x_0}/(e^{x_1 - x_0} + 1) = 1/\left(1 + e^{-(x_1 - x_0)}\right) = e^{x_1}/(e^{x_0} + e^{x_1})\) is the probability of option 1, and similarly \(e^{x_0}/(e^{x_0} + e^{x_1})\) is the probability of option 0.

This form immediately generalizes to more alternatives as the softmax function, which is a vector-valued function whose i-th coordinate is \(e^{x_i} / \sum_{i=0}^n e^{x_i}\).

More subtly, the symmetric form emphasizes interpreting the input x as \(x_1 - x_0\) and thus relative to some reference point, implicitly to \(x_0 = 0\). Notably, the softmax function is invariant under adding a constant to all the logits \(x_i\), which corresponds to the difference \(x_j - x_i\) being the log-odds for option j against option i, but the individual logits \(x_i\) not being log-odds on their own. Often one of the options is used as a reference ("pivot"), and its value fixed as 0, so the other logits are interpreted as odds versus this reference. This is generally done with the first alternative, hence the choice of numbering: \(x_0 = 0\), and then \(x_i = x_i - x_0\) is the log-odds for option i against option 0. Since \(e^0 = 1\), this yields the \(+1\) term in many expressions for the logistic function and generalizations.

In growth modeling, numerous generalizations exist, including the generalized logistic curve, the Gompertz function, the cumulative distribution function of the shifted Gompertz distribution, and the hyperbolastic function of type I.

In statistics, where the logistic function is interpreted as the probability of one of two alternatives, the generalization to three or more alternatives is the softmax function, which is vector-valued, as it gives the probability of each alternative.

A typical application of the logistic equation is a common model of population growth (see also population dynamics), originally due to Pierre-François Verhulst in 1838, where the rate of reproduction is proportional to both the existing population and the amount of available resources, all else being equal. The Verhulst equation was published after Verhulst had read Thomas Malthus' An Essay on the Principle of Population, which describes the Malthusian growth model of simple (unconstrained) exponential growth. Verhulst derived his logistic equation to describe the self-limiting growth of a biological population. The equation was rediscovered in 1911 by A. G. McKendrick for the growth of bacteria in broth and experimentally tested using a technique for nonlinear parameter estimation. The equation is also sometimes called the Verhulst-Pearl equation following its rediscovery in 1920 by Raymond Pearl (1879-1940) and Lowell Reed (1888-1966) of the Johns Hopkins University. Another scientist, Alfred J. Lotka derived the equation again in 1925, calling it the law of population growth.

Letting \(P\) represent population size (\(N\) is often used in ecology instead) and \(t\) represent time, this model is formalized by the differential equation:

\[\frac{dP}{dt}=r P \left(1 - \frac{P}{K}\right),\]

where the constant \(r\) defines the growth rate and \(K\) is the carrying capacity.

In the equation, the early, unimpeded growth rate is modeled by the first term \(+rP\). The value of the rate \(r\) represents the proportional increase of the population \(P\) in one unit of time. Later, as the population grows, the modulus of the second term (which multiplied out is \(-r P^2 / K\)) becomes almost as large as the first, as some members of the population \(P\) interfere with each other by competing for some critical resource, such as food or living space. This antagonistic effect is called the bottleneck, and is modeled by the value of the parameter \(K\). The competition diminishes the combined growth rate, until the value of \(P\) ceases to grow (this is called maturity of the population). The solution to the equation (with \(P_0\) being the initial population) is

\[P(t) = \frac{K P_0 e^{rt}}{K + P_0 \left( e^{rt} - 1\right)} = \frac{K}{1+\left(\frac{K-P_0}{P_0}\right)e^{-rt}},\]

where

Condensed: the full section is in Wikipedia.

Logistic functions are used in several roles in statistics. For example, they are the cumulative distribution function of the logistic family of distributions, and they are, a bit simplified, used to model the chance a chess player has to beat their opponent in the Elo rating system. More specific examples now follow.