Orthogonality

In this section we define a couple more operations with vectors, and prove a few theorems. At first blush these definitions and results will not appear central to what follows, but we will make use of them at key points in the remainder of the course (such as , ). Because we have chosen to use $\complexes$ as our set of scalars, this subsection is a bit more, uh, complex than it would be for the real numbers. We will explain as we go along how things get easier for the real numbers ${\mathbb R}$. If you have not already, now would be a good time to review some of the basic properties of arithmetic with complex numbers described in . With that done, we can extend the basics of complex number arithmetic to our study of vectors in $\complex{m}$.

1. \set{\colvector{1\\-1\\2},\,\colvector{5\\3\\-1},\,\colvector{8\\4\\-2}}
2. What is the distinction between an orthogonal set and an orthonormal set?
3. What is nice about the output of the Gram-Schmidt process?

We know how the addition and multiplication of complex numbers is employed in defining the operations for vectors in $\complex{m}$ ( and ). We can also extend the idea of the conjugate to vectors.

With this definition we can show that the conjugate of a column vector behaves as we would expect with regard to vector addition and scalar multiplication.

These two theorems together tell us how we can push complex conjugation through linear combinations.

This operation is a bit different in that we begin with two vectors but produce a scalar. Computing one is straightforward.

In the case where the entries of our vectors are all real numbers (as in the second part of ), the computation of the inner product may look familiar and be known to you as a dot product or scalar product. So you can view the inner product as a generalization of the scalar product to vectors from $\complex{m}$ (rather than ${\mathbb R}^m$).

Note that we have chosen to conjugate the entries of the first vector listed in the inner product, while it is almost equally feasible to conjugate entries from the second vector instead. In particular, prior to Version 2.90, we did use the latter definition, and this has now changed to the former, with resulting adjustments propogated up through (only). However, conjugating the first vector leads to much nicer formulas for certain matrix decompositions and also shortens some proofs.

There are several quick theorems we can now prove, and they will each be useful later.

Condensed: the full section is in Beezer, A First Course in Linear Algebra.

If treating linear algebra in a more geometric fashion, the length of a vector occurs naturally, and is what you would expect from its name. With complex numbers, we will define a similar function. Recall that if $c$ is a complex number, then $\modulus{c}$ denotes its modulus ().

Computing a norm is also easy to do.

Notice how the norm of a vector with real number entries is just the length of the vector. Inner products and norms are related by the following theorem.

When our vectors have entries only from the real numbers says that the dot product of a vector with itself is equal to the length of the vector squared.

Notice that contains three implications: \vect{u}\in\complex{m}&\Rightarrow\innerproduct{\vect{u}}{\vect{u}}\geq 0\\ \vect{u}=\zerovector&\Rightarrow\innerproduct{\vect{u}}{\vect{u}}=0\\ \innerproduct{\vect{u}}{\vect{u}}=0&\Rightarrow\vect{u}=\zerovector

The results contained in are summarized by saying the inner product is positive definite.

Condensed: the full section is in Beezer, A First Course in Linear Algebra.

Orthogonal is a generalization of perpendicular. You may have used mutually perpendicular vectors in a physics class, or you may recall from a calculus class that perpendicular vectors have a zero dot product. We will now extend these ideas into the realm of higher dimensions and complex scalars.

We extend this definition to whole sets by requiring vectors to be pairwise orthogonal. Despite using the same word, careful thought about what objects you are using will eliminate any source of confusion.

We now define the prototypical orthogonal set, which we will reference repeatedly.

Notice that $\vect{e}_j$ is identical to column $j$ of the $m\times m$ identity matrix $I_m$ () and is a pivot column for $I_m$, since the identity matrix is in reduced row-echelon form. These observations will often be useful. We will reserve the notation $\vect{e}_i$ for these vectors. It is not hard to see that the set of standard unit vectors is an orthogonal set.

Condensed: the full section is in Beezer, A First Course in Linear Algebra.

The Gram-Schmidt Procedure is really a theorem. It says that if we begin with a linearly independent set of $p$ vectors, $S$, then we can do a number of calculations with these vectors and produce an orthogonal set of $p$ vectors, $T$, so that $\spn{S}=\spn{T}$. Given the large number of computations involved, it is indeed a procedure to do all the necessary computations, and it is best employed on a computer. However, it also has value in proofs where we may on occasion wish to replace a linearly independent set by an orthogonal set.

This is our first occasion to use the technique of mathematical induction for a proof, a technique we will see again several times, especially in . So study the simple example described in first.

One final definition related to orthogonal vectors.

Once you have an orthogonal set, it is easy to convert it to an orthonormal set multiply each vector by the reciprocal of its norm, and the resulting vector will have norm 1. This scaling of each vector will not affect the orthogonality properties (apply ).

We will see orthonormal sets again in . They are intimately related to unitary matrices () through . Some of the utility of orthonormal sets is captured by in . Orthonormal sets appear once again in where they are key in orthonormal diagonalization.

Condensed: the full section is in Beezer, A First Course in Linear Algebra.

Orthogonality is a term with various meanings depending on the context.

In mathematics, orthogonality is the generalization of the geometric notion of perpendicularity. Although many authors use the two terms perpendicular and orthogonal interchangeably, the term perpendicular is more specifically used for lines and planes that intersect to form a right angle, whereas orthogonal is used in generalizations, such as orthogonal vectors or orthogonal curves.

The term is also used in other fields like physics, art, computer science, statistics, and economics.

The word comes from the Ancient Greek ὀρθός (orthós), meaning "upright", and γωνία (gōnía), meaning "angle".

The Ancient Greek ὀρθογώνιον (orthogṓnion) and Classical Latin orthogonium originally denoted a rectangle. Later, they came to mean a right triangle. In the 12th century, the post-classical Latin word orthogonalis came to mean a right angle or something related to a right angle.

In mathematics, orthogonality is the generalization of the geometric notion of perpendicularity to linear algebra of bilinear forms.

Two elements u and v of a vector space with bilinear form \(B\) are orthogonal when \(B(\mathbf{u},\mathbf{v})= 0\). Depending on the bilinear form, the vector space may contain null vectors, non-zero self-orthogonal vectors, in which case perpendicularity is replaced with hyperbolic orthogonality.

In the case of function spaces, families of functions are used to form an orthogonal basis, such as in the contexts of orthogonal polynomials, orthogonal functions, and combinatorics.

In optics, polarization states are said to be orthogonal when they propagate independently of each other, as in vertical and horizontal linear polarization or right-handed and left-handed circular polarization.

In special relativity, a time axis determined by a rapidity of motion is hyperbolic-orthogonal to a space axis of simultaneous events, also determined by the rapidity. The theory features relativity of simultaneity.

In quantum mechanics, a sufficient (but not necessary) condition that two eigenstates of a Hermitian operator, \(\psi_m\) and \(\psi_n\), are orthogonal is that they correspond to different eigenvalues. This means, in Dirac notation, that \(\langle \psi_m | \psi_n \rangle = 0\) if \(\psi_m\) and \(\psi_n\) correspond to different eigenvalues. This follows from the fact that Schrödinger's equation is a Sturm-Liouville equation (in Schrödinger's formulation) or that observables are given by Hermitian operators (in Heisenberg's formulation).

In art, the perspective (imaginary) lines pointing to the vanishing point are referred to as "orthogonal lines". The term "orthogonal line" often has a quite different meaning in the literature of modern art criticism. Many works by painters such as Piet Mondrian and Burgoyne Diller are noted for their exclusive use of "orthogonal lines", not, however, with reference to perspective, but rather referring to lines that are straight and exclusively horizontal or vertical, forming right angles where they intersect. For example, an essay of the Thyssen-Bornemisza Museum states that "Mondrian [...] dedicated his entire oeuvre to the investigation of the balance between orthogonal lines and primary colours."

Orthogonality in programming language design is the ability to use various language features in arbitrary combinations with consistent results. This usage was introduced by Van Wijngaarden in the design of Algol 68:

Orthogonality is a system design property which guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system. Typically this is achieved through the separation of concerns and encapsulation, and it is essential for feasible and compact designs of complex systems. The emergent behavior of a system consisting of components should be controlled strictly by formal definitions of its logic and not by side effects resulting from poor integration, i.e., non-orthogonal design of modules and interfaces. Orthogonality reduces testing and development time because it is easier to verify designs that neither cause side effects nor depend on them.

An instruction set is said to be orthogonal if it lacks redundancy (i.e., there is only a single instruction that can be used to accomplish a given task) and is designed such that instructions can use any register in any addressing mode. This terminology results from considering an instruction as a vector whose components are the instruction fields. One field identifies the registers to be operated upon and another specifies the addressing mode. An orthogonal instruction set uniquely encodes all combinations of registers and addressing modes.

In telecommunications, multiple access schemes are orthogonal when an ideal receiver can completely reject arbitrarily strong unwanted signals from the desired signal using different basis functions. One such scheme is time-division multiple access (TDMA), where the orthogonal basis functions are nonoverlapping rectangular pulses ("time slots").

Another scheme is orthogonal frequency-division multiplexing (OFDM), which refers to the use, by a single transmitter, of a set of frequency multiplexed signals with the exact minimum frequency spacing needed to make them orthogonal so that they do not interfere with each other. Well known examples include (a, g, and n) versions of 802.11 Wi-Fi; WiMAX; ITU-T G.hn, DVB-T, the terrestrial digital TV broadcast system used in most of the world outside North America; and DMT (Discrete Multi Tone), the standard form of ADSL.

In OFDM, the subcarrier frequencies are chosen so that the subcarriers are orthogonal to each other, meaning that crosstalk between the subchannels is eliminated and intercarrier guard bands are not required. This greatly simplifies the design of both the transmitter and the receiver. In conventional FDM, a separate filter for each subchannel is required.

When performing statistical analysis, independent variables that affect a particular dependent variable are said to be orthogonal if they are uncorrelated, since the covariance forms an inner product. In this case the same results are obtained for the effect of any of the independent variables upon the dependent variable, regardless of whether one models the effects of the variables individually with simple regression or simultaneously with multiple regression. If correlation is present, the factors are not orthogonal and different results are obtained by the two methods. This usage arises from the fact that if centered by subtracting the expected value (the mean), uncorrelated variables are orthogonal in the geometric sense discussed above, both as observed data (i.e., vectors) and as random variables (i.e., density functions). One econometric formalism that is alternative to the maximum likelihood framework, the Generalized Method of Moments, relies on orthogonality conditions. In particular, the Ordinary Least Squares estimator may be easily derived from an orthogonality condition between the explanatory variables and model residuals.

In taxonomy, an orthogonal classification is one in which no item is a member of more than one group, that is, the classifications are mutually exclusive.

In chemistry and biochemistry, an orthogonal interaction occurs when there are two pairs of substances and each substance can interact with their respective partner, but does not interact with either substance of the other pair. For example, DNA has two orthogonal pairs: cytosine and guanine form a base-pair, and adenine and thymine form another base-pair, but other base-pair combinations are strongly disfavored. As a chemical example, tetrazine reacts with transcyclooctene and azide reacts with cyclooctyne without any cross-reaction, so these are mutually orthogonal reactions, and so, can be performed simultaneously and selectively.