Inner product space

In mathematics, an inner product space is a real or complex vector space endowed with an operation called an inner product. The inner product of two vectors in the space is a scalar, often denoted with angle brackets such as in \(\langle a, b \rangle\). Inner products allow formal definitions of intuitive geometric notions, such as lengths, angles, and orthogonality (zero inner product) of vectors. Inner product spaces generalize Euclidean vector spaces, in which the inner product is the dot product or scalar product of Cartesian coordinates. Inner product spaces of infinite dimensions are widely used in functional analysis. Inner product spaces over the field of complex numbers are sometimes referred to as unitary spaces. The first usage of the concept of a vector space with an inner product is due to Giuseppe Peano, in 1898.

An inner product naturally induces an associated norm, (denoted \(|x|\) and \(|y|\) in the picture); so, every inner product space is a normed vector space. If this normed space is also complete (that is, a Banach space) then the inner product space is a Hilbert space. If an inner product space H is not a Hilbert space, it can be extended by completion to a Hilbert space \(\overline{H}.\) This means that \(H\) is a linear subspace of \(\overline{H},\) the inner product of \(H\) is the restriction of that of \(\overline{H},\) and \(H\) is dense in \(\overline{H}\) for the topology defined by the norm.

In this article, F denotes a field that is either the real numbers \(\R,\) or the complex numbers \(\Complex.\) A scalar is thus an element of F. A bar over an expression representing a scalar denotes the complex conjugate of this scalar. A zero vector is denoted \(\mathbf 0\) for distinguishing it from the scalar 0.

An inner product space is a vector space V over the field F together with an inner product, that is, a map \[\langle \cdot \operatorname{,} \cdot \rangle : V \times V \to F\] that satisfies the following three properties for all vectors \(x,y,z\in V\) and all scalars \(a,b \in F\).

• Conjugate symmetry: \[\langle x, y \rangle = \overline{\langle y, x \rangle}.\] As \(a = \overline{a}\) if and only if \(a\) is real, conjugate symmetry implies that \(\langle x, x \rangle\) is always a real number. If F is \(\R\), conjugate symmetry is just symmetry.
• Linearity in the first argument: \[\langle ax+by, z \rangle = a \langle x, z \rangle + b \langle y, z \rangle.\]
• Positive-definiteness: if \(x\) is not zero, then \[\langle x, x \rangle > 0\] (conjugate symmetry implies that \(\langle x, x \rangle\) is real).

If the positive-definiteness condition is replaced by merely requiring that \(\langle x, x \rangle \geq 0\) for all \(x\), then one obtains the definition of positive semi-definite Hermitian form. A positive semi-definite Hermitian form \(\langle \cdot, \cdot \rangle\) is an inner product if and only if for all \(x\), if \(\langle x, x \rangle = 0\) then \(x = \mathbf 0\).

In the following properties, which result almost immediately from the definition of an inner product, x, y and z are arbitrary vectors, and a and b are arbitrary scalars.

• \(\langle \mathbf{0}, x \rangle=\langle x,\mathbf{0}\rangle=0.\)
• \(\langle x, x \rangle\) is real and nonnegative.
• \(\langle x, x \rangle = 0\) if and only if \(x=\mathbf{0}.\)
• \(\langle x, ay+bz \rangle= \overline a \langle x, y \rangle + \overline b \langle x, z \rangle\), that is conjugate-linearity (for the 2nd argument).
  This implies that an inner product is a sesquilinear form.
• \(\langle x + y, x + y \rangle = \langle x, x \rangle + 2\operatorname{Re}(\langle x, y \rangle) + \langle y, y \rangle,\) where \(\operatorname{Re}\)
  denotes the real part of its argument.

Over \(\R\), conjugate-symmetry reduces to symmetry, and sesquilinearity reduces to bilinearity. Hence an inner product on a real vector space is a positive-definite symmetric bilinear form. The binomial expansion of a square becomes \[\langle x + y, x + y \rangle = \langle x, x \rangle + 2\langle x, y \rangle + \langle y, y \rangle .\]

Several notations are used for inner products, including \(\langle \cdot, \cdot \rangle\), \(\left ( \cdot, \cdot \right )\), \(\langle \cdot | \cdot \rangle\) and \(\left ( \cdot | \cdot \right )\), as well as the usual dot product.

Some authors, especially in physics and matrix algebra, prefer to define inner products and sesquilinear forms with linearity in the second argument rather than the first. Then the first argument becomes conjugate linear, rather than the second. Bra, ket notation in quantum mechanics also uses slightly different notation, i.e. \(\langle \cdot | \cdot \rangle\), where \(\langle x | y \rangle := \left ( y, x \right )\).

Among the simplest examples of inner product spaces are \(\R\) and \(\Complex.\) The real numbers \(\R\) are a vector space over \(\R\) that becomes an inner product space with arithmetic multiplication as its inner product: \[\langle x, y \rangle := x y \quad \text{ for } x, y \in \R.\]

The complex numbers \(\Complex\) are a vector space over \(\Complex\) that becomes an inner product space with the inner product \[\langle x, y \rangle := x \overline{y} \quad \text{ for } x, y \in \Complex.\] Unlike with the real numbers, the assignment \((x, y) \mapsto x y\) does not define a complex inner product on \(\Complex.\)

More generally, the real \(n\)-space \(\R^n\) with the dot product is an inner product space, an example of a Euclidean vector space. \[\left\langle \begin{bmatrix} x_1 \\ \vdots \\ x_n \end{bmatrix}, \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix} \right\rangle = x^\operatorname{T} y = \sum_{i=1}^n x_i y_i = x_1 y_1 + \cdots + x_n y_n,\] where \(x^{\operatorname{T}}\) is the transpose of \(x.\)

A function \(\langle \,\cdot, \cdot\, \rangle : \R^n \times \R^n \to \R\) is an inner product on \(\R^n\) if and only if there exists a symmetric positive-definite matrix \(\mathbf{M}\) such that \(\langle x, y \rangle = x^{\operatorname{T}} \mathbf{M} y\) for all \(x, y \in \R^n.\) If \(\mathbf{M}\) is the identity matrix then \(\langle x, y \rangle = x^{\operatorname{T}} \mathbf{M} y\) is the dot product. For another example, if \(n = 2\) and \(\mathbf{M} = \begin{bmatrix} a & b \\ b & d \end{bmatrix}\) is positive-definite (which happens if and only if \(\det \mathbf{M} = a d - b^2 > 0\) and one/both diagonal elements are positive) then for any \(x := \left[x_1, x_2\right]^{\operatorname{T}}, y := \left[y_1, y_2\right]^{\operatorname{T}} \in \R^2,\) \[\langle x, y \rangle := x^{\operatorname{T}} \mathbf{M} y = \left[x_1, x_2\right] \begin{bmatrix} a & b \\ b & d \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = a x_1 y_1 + b x_1 y_2 + b x_2 y_1 + d x_2 y_2.\] As mentioned earlier, every inner product on \(\R^2\) is of this form (where \(b \in \R, a > 0\) and \(d > 0\) satisfy \(a d > b^2\)).

The general form of an inner product on \(\Complex^n\) is known as the Hermitian form and is given by \[\langle x, y \rangle = y^\dagger \mathbf{M} x = \overline{x^\dagger \mathbf{M} y},\] where \(M\) is any Hermitian positive-definite matrix and \(y^{\dagger}\) is the conjugate transpose of \(y.\) For the real case, this corresponds to the dot product of the results of directionally-different scaling of the two vectors, with positive scale factors and orthogonal directions of scaling. It is a weighted-sum version of the dot product with positive weights, up to an orthogonal transformation.

The article on Hilbert spaces has several examples of inner product spaces, wherein the metric induced by the inner product yields a complete metric space. An example of an inner product space which induces an incomplete metric is the space \(C([a, b])\) of continuous complex valued functions \(f\) and \(g\) on the interval \([a, b].\) The inner product is \[\langle f, g \rangle = \int_a^b f(t) \overline{g(t)} \, \mathrm{d}t.\] This space is not complete; consider for example, for the interval [−1, 1] the sequence of continuous "step" functions, \(\{ f_k \}_k,\) defined by: \[f_k(t) = \begin{cases} 0 & t \in [-1, 0] \\ 1 & t \in \left[\tfrac{1}{k}, 1\right] \\ kt & t \in \left(0, \tfrac{1}{k}\right) \end{cases}\]

This sequence is a Cauchy sequence for the norm induced by the preceding inner product, which does not converge to a continuous function.

For real random variables \(X\) and \(Y,\) the expected value of their product \[\langle X, Y \rangle = \mathbb{E}[XY]\] is an inner product. In this case, \(\langle X, X \rangle = 0\) if and only if \(\mathbb{P}[X = 0] = 1\) (that is, \(X = 0\) almost surely), where \(\mathbb{P}\) denotes the probability of the event. This definition of expectation as inner product can be extended to random vectors as well.

The inner product for complex square matrices of the same size is the Frobenius inner product \(\langle A, B \rangle := \operatorname{tr}\left(AB^\dagger\right)\). Since trace and transposition are linear and the conjugation is on the second matrix, it is a sesquilinear operator. We further get Hermitian symmetry by, \[\langle A, B \rangle = \operatorname{tr}\left(AB^\dagger\right) = \overline{\operatorname{tr}\left(BA^\dagger\right)} = \overline{\left\langle B,A \right\rangle}\] Finally, since for \(A\) nonzero, \(\langle A, A\rangle = \sum_{ij} \left|A_{ij}\right|^2 > 0\), we get that the Frobenius inner product is positive definite too, and so is an inner product.

On an inner product space, or more generally a vector space with a nondegenerate form (hence an isomorphism \(V \to V^*\)), vectors can be sent to covectors (in coordinates, via transpose), so that one can take the inner product and outer product of two vectors, not simply of a vector and a covector.

Every inner product space induces a norm, called its canonical norm, that is defined by \[\|x\| = \sqrt{\langle x, x \rangle}.\] With this norm, every inner product space becomes a normed vector space.

So, every general property of normed vector spaces applies to inner product spaces. In particular, one has the following properties:

Absolute homogeneity

\[\|ax\| = |a| \, \|x\|\] for every \(x \in V\) and \(a \in F\) (this results from \(\langle ax, ax \rangle = a\overline a \langle x, x \rangle\)).

Triangle inequality

\[\|x + y\| \leq \|x\| + \|y\|\] for \(x, y\in V.\) These two properties show that one has indeed a norm.

Cauchy-Schwarz inequality

\[|\langle x, y \rangle| \leq \|x\| \, \|y\|\] for every \(x, y\in V,\) with equality if and only if \(x\) and \(y\) are linearly dependent.

Parallelogram law

\[\|x + y\|^2 + \|x - y\|^2 = 2\|x\|^2 + 2\|y\|^2\] for every \(x, y\in V.\) The parallelogram law is a necessary and sufficient condition for a norm to be defined by an inner product.

Polarization identity

\[\|x + y\|^2 = \|x\|^2 + \|y\|^2 + 2\operatorname{Re}\langle x, y \rangle\] for every \(x, y\in V.\) The inner product can be retrieved from the norm by the polarization identity, since its imaginary part is the real part of \(\langle x, iy \rangle.\)

Ptolemy's inequality

\[\|x - y\| \, \|z\| ~+~ \|y - z\| \, \|x\| ~\geq~ \|x - z\| \, \|y\|\] for every \(x, y,z\in V.\) Ptolemy's inequality is a necessary and sufficient condition for a seminorm to be the norm defined by an inner product.

Orthogonality

Two vectors \(x\) and \(y\) are said to be orthogonal, often written \(x \perp y,\) if their inner product is zero, that is, if \(\langle x, y \rangle = 0.\)
This happens if and only if \(\|x\| \leq \|x + s y\|\) for all scalars \(s,\) and if and only if the real-valued function \(f(s) := \|x + s y\|^2 - \|x\|^2\) is non-negative. (This is a consequence of the fact that, if \(y \neq 0\) then the scalar \(s_0 = - \tfrac{\overline{\langle x, y \rangle}}{\|y\|^2}\) minimizes \(f\) with value \(f\left(s_0\right) = - \tfrac{|\langle x, y \rangle|^2}{\|y\|^2},\) which is always non positive).
For a complex inner product space \(H,\) a linear operator \(T : V \to V\) is identically \(0\) if and only if \(x \perp T x\) for every \(x \in V.\) This is not true in general for real inner product spaces, as it is a consequence of conjugate symmetry being distinct from symmetry for complex inner products. A counterexample in a real inner product space is \(T\) a 90° rotation in \(\mathbb{R}^2\), which maps every vector to an orthogonal vector but is not identically \(0\).

Orthogonal complement

The orthogonal complement of a subset \(C \subseteq V\) is the set \(C^{\bot}\) of the vectors that are orthogonal to all elements of C; that is, \[C^{\bot} := \{\,y \in V : \langle y, c \rangle = 0 \text{ for all } c \in C\,\}.\] This set \(C^{\bot}\) is always a closed vector subspace of \(V\) and if the closure \(\operatorname{cl}_V C\) of \(C\) in \(V\) is a vector subspace then \(\operatorname{cl}_V C = \left(C^{\bot}\right)^{\bot}.\)

Pythagorean theorem

If \(x\) and \(y\) are orthogonal, then \[\|x\|^2 + \|y\|^2 = \|x + y\|^2.\] This may be proved by expressing the squared norms in terms of the inner products, using additivity for expanding the right-hand side of the equation.
The name Pythagorean theorem arises from the geometric interpretation in Euclidean geometry.

Parseval's identity

An induction on the Pythagorean theorem yields: if \(x_1, \ldots, x_n\) are pairwise orthogonal, then \[\sum_{i=1}^n \|x_i\|^2 = \left\|\sum_{i=1}^n x_i\right\|^2.\]

Angle

When \(\langle x, y \rangle\) is a real number then the Cauchy-Schwarz inequality implies that \(\frac{\langle x, y \rangle}{\|x\| \, \|y\|} \in [-1, 1],\) and thus that \[\angle(x, y) = \arccos \frac{\langle x, y \rangle}{\|x\| \, \|y\|},\] is a real number. This allows defining the (non oriented) angle of two vectors in modern definitions of Euclidean geometry in terms of linear algebra. This is also used in data analysis, under the name "cosine similarity", for comparing two vectors of data. Furthermore, if \(\langle x, y \rangle\) is negative, the angle \(\angle(x, y)\) is larger than 90 degrees. This property is often used in computer graphics (e.g., in back-face culling) to analyze a direction without having to evaluate trigonometric functions.