Statistical hypothesis test

A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. Then a decision is made, either by comparing the test statistic to a critical value or equivalently by evaluating a p-value computed from the test statistic. Roughly 100 specialized statistical tests are in use.

The goal of a hypothesis test is to establish whether certain properties of a statistical population are true by examining sample data. Typically, the population is modelled by a random variable whose distribution has unknown parameters. For example, a medical trial may wish to establish whether a particular drug is effective in treating high blood pressure, with "the change in blood pressure observed in a patient who takes the drug" being the random variable. An example hypothesis could be "the mean change in blood pressure is zero" or "the mean change in blood pressure is negative". In general, any statement about the parameters describing a population can be a hypothesis (but not a statement about the sample).

The test compares two hypotheses: a default null hypothesis (denoted H₀) and its negation, the alternative hypothesis (H₁). It is usually consistent with the research hypothesis because it is constructed from literature review, previous studies, etc. However, the research hypothesis is sometimes consistent with the null hypothesis. The null hypothesis and alternative hypothesis are two mutually exclusive statements.

"The statement being tested in a test of statistical significance is called the 'null hypothesis'. The test of significance is designed to assess the strength of the evidence against the null hypothesis. Usually, the null hypothesis is a statement of 'no effect' or 'no difference'."

If the sample data are consistent with the null hypothesis, then you do not reject the null hypothesis; if the sample data are inconsistent with the null hypothesis, then you reject the null hypothesis and conclude that the alternative hypothesis is true.

Typically the test will select a null hypothesis that the intervention being studied has no effect, or that the population parameter takes some "obvious" value. A test statistic is computed from the given sample data, and the tester calculates the conditional probability of observing a value at least this extreme, supposing the null hypothesis is true. If this probability (called the p-value) is less than the significance level of the test (denoted \(\alpha\)), then the null hypothesis is rejected. The test does not conclude that the null hypothesis is false, or that the probability that the null hypothesis is false is less than \(\alpha\).

Because it is usually impossible to definitely establish whether the hypothesis being tested is true or false from a sample, the conclusion of a hypothesis test is not certain to be correct. There are two possible classes of error:

• A type I error, in which the null hypothesis is rejected despite the null hypothesis being true, with probability \(\alpha = P(\text{reject } H_0|H_0)\). This is the same as the significance level of the test.
• A type II error, in which the null hypothesis is accepted despite the alternative hypothesis being true, with probability \(\beta = P(\text{accept }H_0|H_1)\). The quantity \(1-\beta\) is called the power of the test.

Some further definitions:

Condensed: the full section is in Wikipedia.

The history of the null and alternative hypotheses has much to do with the history of statistical tests. While hypothesis testing was popularized early in the 20th century, early forms were used in the 1700s. The first use is credited to John Arbuthnot (1710), followed by Pierre-Simon Laplace (1770s), in analyzing the human sex ratio at birth; see § Human sex ratio.

1778: Pierre Laplace compares the birthrates of boys and girls in multiple European cities. He states: "it is natural to conclude that these possibilities are very nearly in the same ratio". Thus, the null hypothesis in this case that the birthrates of boys and girls should be equal given "conventional wisdom".

1900: Karl Pearson develops the chi squared test to determine "whether a given form of frequency curve will effectively describe the samples drawn from a given population." Thus the null hypothesis is that a population is described by some distribution predicted by theory. He uses as an example the numbers of five and sixes in the Weldon dice throw data.

1904: Karl Pearson develops the concept of "contingency" in order to determine whether outcomes are independent of a given categorical factor. Here the null hypothesis is by default that two things are unrelated (e.g. scar formation and death rates from smallpox). The null hypothesis in this case is no longer predicted by theory or conventional wisdom, but is instead the principle of indifference that led Fisher and others to dismiss the use of "inverse probabilities".

The concept of an alternative hypothesis in testing was devised by Jerzy Neyman and Egon Pearson, and it is used in the Neyman-Pearson lemma. It forms a major component in modern statistical hypothesis testing. However it was not part of Ronald Fisher's formulation of statistical hypothesis testing, and he opposed its use. In Fisher's approach to testing, the central idea is to assess whether the observed dataset could have resulted from chance if the null hypothesis were assumed to hold, notionally without preconceptions about what other models might hold. Modern statistical hypothesis testing accommodates this type of test since the alternative hypothesis can be just the negation of the null hypothesis.

Modern significance testing is largely the product of Karl Pearson (p-value, Pearson's chi-squared test), William Sealy Gosset (Student's t-distribution), and Ronald Fisher ("null hypothesis", analysis of variance, "significance test"), while hypothesis testing was developed by Jerzy Neyman and Egon Pearson (son of Karl). Ronald Fisher began his life in statistics as a Bayesian (Zabell 1992), but Fisher soon grew disenchanted with the subjectivity involved (namely use of the principle of indifference when determining prior probabilities), and sought to provide a more "objective" approach to inductive inference.

Fisher emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an inconsistent hybrid of the Fisher vs Neyman/Pearson formulation, methods and terminology developed in the early 20th century.

Fisher popularized the "significance test". He required a null-hypothesis (corresponding to a population frequency distribution) and a sample. His (now familiar) calculations determined whether to reject the null-hypothesis or not. Significance testing did not utilize an alternative hypothesis so there was no concept of a Type II error (false negative).

The p-value was devised as an informal, but objective, index meant to help a researcher determine (based on other knowledge) whether to modify future experiments or strengthen one's faith in the null hypothesis. Hypothesis testing (and Type I/II errors) was devised by Neyman and Pearson as a more objective alternative to Fisher's p-value, also meant to determine researcher behaviour, but without requiring any inductive inference by the researcher.

Neyman & Pearson considered a different problem to Fisher (which they called "hypothesis testing"). They initially considered two simple hypotheses (both with frequency distributions). They calculated two probabilities and typically selected the hypothesis associated with the higher probability (the hypothesis more likely to have generated the sample). Their method always selected a hypothesis. It also allowed the calculation of both types of error probabilities. In a series of papers (published over a decade starting in 1928) Neyman & Pearson defined the statistical hypothesis test as a proposed improvement on Fisher's test. The papers provided much of the terminology for statistical tests including alternative hypothesis and H₀ as a hypothesis to be tested using observational data (with H₁, H₂... as alternatives).

The dispute between Fisher and Neyman-Pearson was waged on philosophical grounds, characterized by a philosopher as a dispute over the proper role of models in statistical inference.

Condensed: the full section is in Wikipedia.

The modern version of hypothesis testing is generally called the null hypothesis significance testing (NHST) and is a hybrid of the Fisher approach with the Neyman-Pearson approach. In 2000, Raymond S. Nickerson wrote an article stating that NHST was (at the time) "arguably the most widely used method of analysis of data collected in psychological experiments and has been so for about 70 years" and that it was at the same time "very controversial".

This fusion resulted from confusion by writers of statistical textbooks (as predicted by Fisher) beginning in the 1940s (but signal detection, for example, still uses the Neyman/Pearson formulation). Great conceptual differences and many caveats in addition to those mentioned above were ignored. Neyman and Pearson provided the stronger terminology, the more rigorous mathematics and the more consistent philosophy, but the subject taught today in introductory statistics has more similarities with Fisher's method than theirs.

Sometime around 1940, authors of statistical text books began combining the two approaches by using the p-value in place of the test statistic (or data) to test against the Neyman-Pearson "significance level".

The null hypothesis is a default hypothesis that a quantity to be measured is zero (null). Typically, the quantity to be measured is the difference between two situations. For instance, trying to determine if there is a positive proof that an effect has occurred or that samples derive from different batches.

The null hypothesis is generally assumed to remain possibly true. Multiple analyses can be performed to show how the hypothesis should either be rejected or excluded e.g. having a high confidence level, thus demonstrating a statistically significant difference. This is demonstrated by showing that zero is outside of the specified confidence interval of the measurement on either side, typically within the real numbers. Failure to exclude the null hypothesis (with any confidence) does not logically confirm or support the (unprovable) null hypothesis. (When it is not proven that something is e.g. bigger than x, it does not necessarily imply it is smaller or equal than x; it may instead be a poor quality measurement with low accuracy. Confirming the null hypothesis is two-sided would amount to positively proving it is bigger or equal than 0 and to positively proving it is smaller or equal than 0; this is something for which infinite accuracy is needed as well as exactly zero effect, neither of which normally are realistic. Also, measurements never indicate a non-zero probability of exactly zero difference.) So failure of an exclusion of a null hypothesis amounts to a "don't know" at the specified confidence level; it does not immediately imply null somehow, as the data may already show a (less strong) indication for a non-null. The used confidence level does absolutely certainly not correspond to the likelihood of null at failing to exclude; in fact in this case a high used confidence level expands the still plausible range.

A non-null hypothesis can have the following meanings, depending on the author a) a value other than zero is used, b) some margin other than zero is used and c) the "alternative" hypothesis.

Testing (excluding or failing to exclude) the null hypothesis provides evidence that there are (or are not) statistically sufficient grounds to believe there is a relationship between two phenomena (e.g., that a potential treatment has a non-zero effect, either way). Testing the null hypothesis is a central task in statistical hypothesis testing in the modern practice of science. There are precise criteria for excluding or not excluding a null hypothesis at a certain confidence level. The confidence level should indicate the likelihood that much more and better data would still be able to exclude the null hypothesis on the same side.

In the hypothesis testing approach of Jerzy Neyman and Egon Pearson, a null hypothesis is contrasted with an alternative hypothesis, and the two hypotheses are distinguished on the basis of data, with certain error rates. It is used in formulating answers in research.

Condensed: the full section is in Wikipedia.

Paul Meehl has argued that the epistemological importance of the choice of null hypothesis has gone largely unacknowledged. When the null hypothesis is predicted by theory, a more precise experiment will be a more severe test of the underlying theory. When the null hypothesis defaults to "no difference" or "no effect", a more precise experiment is a less severe test of the theory that motivated performing the experiment.

Fisher and Neyman opposed the subjectivity of probability. Their views contributed to the objective definitions. The core of their historical disagreement was philosophical.

Many of the philosophical criticisms of hypothesis testing are discussed by statisticians in other contexts, particularly correlation does not imply causation and the design of experiments. Hypothesis testing is of continuing interest to philosophers.

Statistics is increasingly being taught in schools with hypothesis testing being one of the elements taught. Many conclusions reported in the popular press (political opinion polls to medical studies) are based on statistics. Some writers have stated that statistical analysis of this kind allows for thinking clearly about problems involving mass data, as well as the effective reporting of trends and inferences from said data, but caution that writers for a broad public should have a solid understanding of the field in order to use the terms and concepts correctly. An introductory college statistics class places much emphasis on hypothesis testing, perhaps half of the course. Such fields as literature and divinity now include findings based on statistical analysis (see the Bible Analyzer). An introductory statistics class teaches hypothesis testing as a cookbook process. Hypothesis testing is also taught at the postgraduate level. Statisticians learn how to create good statistical test procedures (like z, Student's t, F and chi-squared). Statistical hypothesis testing is considered a mature area within statistics, but a limited amount of development continues.

An academic study states that the cookbook method of teaching introductory statistics leaves no time for history, philosophy or controversy. Hypothesis testing has been taught as received unified method. Surveys showed that graduates of the class were filled with philosophical misconceptions (on all aspects of statistical inference) that persisted among instructors. While the problem was addressed more than a decade ago, and calls for educational reform continue, students still graduate from statistics classes holding fundamental misconceptions about hypothesis testing. Ideas for improving the teaching of hypothesis testing include encouraging students to search for statistical errors in published papers, teaching the history of statistics and emphasizing the controversy in a generally dry subject.

Raymond S. Nickerson commented:

Hypothesis testing requires constructing a statistical model of what the data would look like if chance or random processes alone were responsible for the results. The hypothesis that chance alone is responsible for the results is called the null hypothesis. The model of the result of the random process is called the distribution under the null hypothesis. The obtained results are compared with the distribution under the null hypothesis, and the likelihood of finding the obtained results is thereby determined.

Hypothesis testing works by collecting data and measuring how likely the particular set of data is (assuming the null hypothesis is true), when the study is on a randomly selected representative sample. The null hypothesis assumes no relationship between variables in the population from which the sample is selected.

If the data-set of a randomly selected representative sample is very unlikely relative to the null hypothesis (defined as being part of a class of sets of data that are rarely observed), the experimenter rejects the null hypothesis, concluding it (probably) is false. This class of data-sets is usually specified via a test statistic, which is designed to measure the extent of apparent departure from the null hypothesis. The procedure works by assessing whether the observed departure, measured by the test statistic, is larger than a value defined, so that the probability of occurrence of a more extreme value is small under the null hypothesis (usually in less than either 5% or 1% of similar data-sets in which the null hypothesis does hold).

If the data do not contradict the null hypothesis, then only a weak conclusion can be made: namely, that the observed data set provides insufficient evidence against the null hypothesis. In this case, because the null hypothesis could be true or false, in some contexts this is interpreted as meaning that the data give insufficient evidence to make any conclusion, while in other contexts, it is interpreted as meaning that there is not sufficient evidence to support changing from a currently useful regime to a different one. Nevertheless, if at this point the effect appears likely and/or large enough, there may be an incentive to further investigate, such as running a bigger sample.

For instance, a certain drug may reduce the risk of having a heart attack. Possible null hypotheses are "this drug does not reduce the risk of having a heart attack" or "this drug has no effect on the risk of having a heart attack". The test of the hypothesis consists of administering the drug to half of the people in a study group as a controlled experiment. If the data show a statistically significant change in the people receiving the drug, the null hypothesis is rejected.

There are many types of significance tests for one, two or more samples, for means, variances and proportions, paired or unpaired data, for different distributions, for large and small samples; all have null hypotheses. There are also at least four goals of null hypotheses for significance tests:

• Technical null hypotheses are used to verify statistical assumptions. For example, the residuals between the data and a statistical model cannot be distinguished from random noise. If true, there is no justification for complicating the model.
• Scientific null assumptions are used to directly advance a theory. For example, the angular momentum of the universe is zero. If not true, the theory of the early universe may need revision.
• Null hypotheses of homogeneity are used to verify that multiple experiments are producing consistent results. For example, the effect of a medication on the elderly is consistent with that of the general adult population. If true, this strengthens the general effectiveness conclusion and simplifies recommendations for use.
• Null hypotheses that assert the equality of effect of two or more alternative treatments, for example, a drug and a placebo, are used to reduce scientific claims based on statistical noise. This is the most popular null hypothesis; It is so popular that many statements about significant testing assume such null hypotheses.

Rejection of the null hypothesis is not necessarily the real goal of a significance tester. An adequate statistical model may be associated with a failure to reject the null; the model is adjusted until the null is not rejected. The numerous uses of significance testing were well known to Fisher who discussed many in his book written a decade before defining the null hypothesis.

A statistical significance test shares much mathematics with a confidence interval. They are mutually illuminating. A result is often significant when there is confidence in the sign of a relationship (the interval does not include 0). Whenever the sign of a relationship is important, statistical significance is a worthy goal. This also reveals weaknesses of significance testing: A result can be significant without a good estimate of the strength of a relationship; significance can be a modest goal. A weak relationship can also achieve significance with enough data. Reporting both significance and confidence intervals is commonly recommended.

The varied uses of significance tests reduce the number of generalizations that can be made about all applications.

The choice of the null hypothesis is associated with sparse and inconsistent advice. Fisher mentioned few constraints on the choice and stated that many null hypotheses should be considered and that many tests are possible for each. The variety of applications and the diversity of goals suggests that the choice can be complicated. In many applications the formulation of the test is traditional. A familiarity with the range of tests available may suggest a particular null hypothesis and test. Formulating the null hypothesis is not automated (though the calculations of significance testing usually are). David Cox said, "How [the] translation from subject-matter problem to statistical model is done is often the most critical part of an analysis".

A statistical significance test is intended to test a hypothesis. If the hypothesis summarizes a set of data, there is no value in testing the hypothesis on that set of data. Example: If a study of last year's weather reports indicates that rain in a region falls primarily on weekends, it is only valid to test that null hypothesis on weather reports from any other year. Testing hypotheses suggested by the data is circular reasoning that proves nothing; It is a special limitation on the choice of the null hypothesis.

A routine procedure is as follows: Start from the scientific hypothesis. Translate this to a statistical alternative hypothesis and proceed: "Because H_a expresses the effect that we wish to find evidence for, we often begin with H_a and then set up H₀ as the statement that the hoped-for effect is not present." This advice is reversed for modeling applications where we hope not to find evidence against the null.

A complex case example is as follows: The gold standard in clinical research is the randomized placebo-controlled double-blind clinical trial. But testing a new drug against a (medically ineffective) placebo may be unethical for a serious illness. Testing a new drug against an older medically effective drug raises fundamental philosophical issues regarding the goal of the test and the motivation of the experimenters. The standard "no difference" null hypothesis may reward the pharmaceutical company for gathering inadequate data. "Difference" is a better null hypothesis in this case, but statistical significance is not an adequate criterion for reaching a nuanced conclusion that requires a good numeric estimate of the drug's effectiveness. A "minor" or "simple" proposed change in the null hypothesis ((new vs old) rather than (new vs placebo)) can have a dramatic effect on the utility of a test for complex non-statistical reasons.

The typical steps involved in performing a frequentist hypothesis test in practice are:

1. Define a hypothesis (claim which is testable using data).
2. Select a relevant statistical test with associated test statistic T.
3. Derive the distribution of the test statistic under the null hypothesis from the assumptions. In standard cases this will be a well-known result. For example, the test statistic might follow a Student's t distribution with known degrees of freedom, or a normal distribution with known mean and variance.
4. Select a significance level (α), the maximum acceptable false positive rate. Common values are 5% and 1%.
5. Compute from the observations the observed value t_obs of the test statistic T.
6. Decide to either reject the null hypothesis in favor of the alternative or not reject it. The Neyman-Pearson decision rule is to reject the null hypothesis H₀ if the observed value t_obs is in the critical region, and not to reject the null hypothesis otherwise.

The difference in the two processes applied to the radioactive suitcase example (below):

• "The Geiger-counter reading is 10. The limit is 9. Check the suitcase."
• "The Geiger-counter reading is high; 97% of safe suitcases have lower readings. The limit is 95%. Check the suitcase."

The former report is adequate, the latter gives a more detailed explanation of the data and the reason why the suitcase is being checked.

Not rejecting the null hypothesis does not mean the null hypothesis is "accepted" per se (though Neyman and Pearson used that word in their original writings; see the Interpretation section).

The processes described here are perfectly adequate for computation. They seriously neglect the design of experiments considerations.

It is particularly critical that appropriate sample sizes be estimated before conducting the experiment.

The phrase "test of significance" was coined by statistician Ronald Fisher.

When the null hypothesis is true and statistical assumptions are met, the probability that the p-value will be less than or equal to the significance level \(\alpha\) is at most \(\alpha\). This ensures that the hypothesis test maintains its specified false positive rate (provided that statistical assumptions are met).

The p-value is the probability that a test statistic which is at least as extreme as the one obtained would occur under the null hypothesis. At a significance level of 0.05, a fair coin would be expected to (incorrectly) reject the null hypothesis (that it is fair) in 1 out of 20 tests on average. The p-value does not provide the probability that either the null hypothesis or its opposite is correct (a common source of confusion).

If the p-value is less than the chosen significance threshold (equivalently, if the observed test statistic is in the critical region), then we say the null hypothesis is rejected at the chosen level of significance. If the p-value is not less than the chosen significance threshold (equivalently, if the observed test statistic is outside the critical region), then the null hypothesis is not rejected at the chosen level of significance.

In the "lady tasting tea" example (below), Fisher required the lady to properly categorize all of the cups of tea to justify the conclusion that the result was unlikely to result from chance. His test revealed that if the lady was effectively guessing at random (the null hypothesis), there was a 1.4% chance that the observed results (perfectly ordered tea) would occur.