Linear regression and correlation

Linear regression for two variables is based on a linear equation with one independent variable. The equation has the form

where a and b are constant numbers.

The variable x is the independent variable; y is the dependent variable. Typically, you choose a value to substitute for the independent variable and then solve for the dependent variable.

The graph of a linear equation of the form y = a + bx is a straight line. Any line that is not vertical can be described by this equation.

For the linear equation y = a + bx, b = slope and a = y-inttercept. From algebra, recall that the slope is a number that describes the steepness of a line; the y-intercept is the y-coordinate of the point (0, a), where the line crosses the y-axis.

Please note that in previous courses you learned \(y=mx+b\) was the slope-intercept form of the equation, where m represented the slope and b represented the y-intercept. In this text, the form \(y=a+bx\) is used, where a is the y-intercept and b is the slope. The key is remembering the coefficient of x is the slope, and the constant number is the y-intercept.

Use the following information to answer the next three exercises. A vacation resort rents scuba equipment to certified divers. The resort charges an up-front fee of $25 and another fee of $12.50 an hour.

Use the following information to answer the next two exercises. A credit card company charges $10 when a payment is late and $5 a day each day the payment remains unpaid.

Use the following information to answer the next exercise. contains real data for the first two decades of flu reporting.

Use the following information to answer the next two exercises. A specialty cleaning company charges an equipment fee and an hourly labor fee. A linear equation that expresses the total amount of the fee the company charges for each session is y = 50 + 100x.

Use the following information to answer the next three questions. As a result of erosion, a river shoreline is losing several thousand pounds of soil each year. A linear equation that expresses the total amount of soil lost per year is y = 12,000x.

Use the following information to answer the next two exercises. The price of a single issue of stock can fluctuate throughout the day. A linear equation that represents the price of stock for Shipment Express is y = 15 – 1.5x, where x is the number of hours passed in an eight-hour day of trading.

Data rarely fit a straight line exactly. Usually, you must be satisfied with rough predictions. Typically, you have a set of data with a scatter plot that appear to fit a straight line. This is called a line of best fit or least-squares regression line.

The third exam score, x, is the independent variable, and the final exam score, y, is the dependent variable. We will plot a regression line that best fits the data. If each of you were to fit a line by eye, you would draw different lines. We can obtain a line of best fit using either the median-–median line approach or by calculating the least-squares regression line.

The ordered pairs are (66.5, 143), (69, 159), and (71, 174).

Condensed — the full section is in OpenStax Statistics.

The process of fitting the best-fit line is called linear regression. We assume that the data are scattered about a straight line. To find that line, we minimize the sum of the squared errors (SSE), or make it as small as possible. Any other line you might choose would have a higher SSE than the best-fit line. This best-fit line is called the least-squares regression line.

The graph of the line of best fit for the third exam/final exam example is as follows:

The least-squares regression line (best-fit line) for the third exam/final exam example has the equation

The y-intercept, a, of the line describes where the plot line crosses the y-axis. The y-intercept of the best-fit line tells us the best value of the relationship when x is zero. In some cases, it does not make sense to figure out what y is when x = 0. For example, in the third exam vs. final exam example, the y-intercept occurs when the third exam score, or x, is zero. Since all the scores are grouped around a passing grade, there is no need to figure out what the final exam score, or y, would be when the third exam was zero.

However, the y-intercept is very useful in many cases. For many examples in science, the y-intercept gives the baseline reading when the experimental conditions aren’'t applied to an experimental system. This baseline indicates how much the experimental condition affects the system. It could also be used to ensure that equipment and measurements are calibrated properly before starting the experiment.

In biology, the concentration of proteins in a sample can be measured using a chemical assay that changes color depending on how much protein is present. The more protein present, the darker the color. The amount of color can be measured by the absorbance reading. shows the expected absorbance readings at different protein concentrations. This is called a standard curve for the assay.

The scatter plot includes the line of best fit.

The y-intercept of this line occurs at 0.0226 mAU. This means the assay gives a reading of 0.0226 mAU when there is no protein present. That is, it is the baseline reading that can be attributed to something else, which, in this case, is some other non-protein chemicals that are absorbing light. We can tell that this line of best fit is reasonable because the y-intercept is small, close to zero. When there is no protein present in the sample, we expect the absorbance to be very small, or close to zero, as well.

The slope of the line, b, describes how changes in the variables are related. It is important to interpret the slope of the line in the context of the situation represented by the data. You should be able to write a sentence interpreting the slope in plain English.

Interpretation of the Slope: The slope of the best-fit line tells us how the dependent variable (y) changes for every one unit increase in the independent (x) variable, on average.

Slope: The slope of the line is b = 4.83.
Interpretation: For a 1-point increase in the score on the third exam, the final exam score increases by 4.83 points, on average.

Condensed — the full section is in OpenStax Statistics.

Besides looking at the scatter plot and seeing that a line seems reasonable, how can you determine whether the line is a good predictor? Use the correlation coefficient as another indicator (besides the scatter plot) of the strength of the relationship between x and y.

The correlation coefficient, r, developed by Karl Pearson during the early 1900s, is numeric and provides a measure of the strength and direction of the linear association between the independent variable x and the dependent variable y.

If you suspect a linear relationship between x and y, then r can measure the strength of the linear relationship.

• The value of r is always between –1 and +1. In other words, –1 ≤ r ≤ 1.
• The size of the correlation r indicates the strength of the linear relationship between x and y. Values of r close to –1 or to +1 indicate a stronger linear relationship between x and y.
• If r = 0, there is absolutely no linear relationship between x and y (no linear correlation).
• If r = 1, there is perfect positive correlation. If r = –1, there is perfect negative correlation. In both these cases, all the original data points lie on a straight line. Of course, in the real world, this does not generally happen.

• A positive value of r means that when x increases, y tends to increase and when x decreases, y tends to decrease (positive correlation).
• A negative value of r means that when x increases, y tends to decrease and when x decreases, y tends to increase (negative correlation).
• The sign of r is the same as the sign of the slope, b, of the best-fit line.

The correlation coefficient is calculated as the quantity of data points times the sum of the quantity of the x-coordinates times the y-coordinates, minus the quantity of the sum of the x-coordinates times the sum of the y-coordinates, all divided by the square root of the quantity of data points times the sum of the x-coordinates squared minus the square of the sum of the x-coordinates, times the number of data points times the sum of the y-coordinates squared minus the square of the sum of the y-coordinates. It can be summarized by the following equation:

\[r=\frac{n\Sigma (xy)-(\Sigma x)(\Sigma y)}{\sqrt{[n\Sigma {x}^{2}-{(\Sigma x)}^{2}][n\Sigma {y}^{2}-{(\Sigma y)}^{2}]}}\]

Condensed — the full section is in OpenStax Statistics.

The variable r² is called the coefficient of determination and it is the square of the correlation coefficient, but it is usually stated as a percentage, rather than in decimal form. It has an interpretation in the context of the data:

• \({r}^{2}\text{,}\) when expressed as a percent, represents the percentage of variation in the dependent (predicted) variable y that can be explained by variation in the independent (explanatory) variable x using the regression (best-fit) line.
• 1 – \({r}^{2}\text{,}\) when expressed as a percentage, represents the percentage of variation in y that is not explained by variation in x using the regression line. This can be seen as the scattering of the observed data points about the regression line.

Consider the third exam/final exam example introduced in the previous section.

• The line of best fit is: ŷ = –173.51 + 4.83x.
• The correlation coefficient is r = .6631.
• The coefficient of determination is r² = .6631² = .4397.

Interpret r² in the context of this example.

• Approximately 44 percent of the variation (0.4397 is approximately 0.44) in the final exam grades can be explained by the variation in the grades on the third exam, using the best-fit regression line.
• Therefore, the rest of the variation (1 – 0.44 = 0.56 or 56 percent) in the final exam grades cannot be explained by the variation of the grades on the third exam with the best-fit regression line. These are the variation of the points that are not as close to the regression line as others.

Use the following information to answer the next five exercises. A random sample of 10 professional athletes produced the following data, where x is the number of endorsements the player has and y is the amount of money made, in millions of dollars.

The correlation coefficient, r, tells us about the strength and direction of the linear relationship between x and y. However, the reliability of the linear model also depends on how many observed data points are in the sample. We need to look at both the correlation coefficient r and the sample size n, together.

We perform a hypothesis test of the significance of the correlation coefficient to decide whether the linear relationship in the sample data is strong enough to use to model the relationship in the population.

The sample data are used to compute r, the correlation coefficient for the sample. If we had data for the entire population, we could find the population correlation coefficient. But, because we have only sample data, we cannot calculate the population correlation coefficient. The sample correlation coefficient, r, is our estimate of the unknown population correlation coefficient.

• The symbol for the population correlation coefficient is ρ, the Greek letter rho.
• ρ = population correlation coefficient (unknown).
• r = sample correlation coefficient (known; calculated from sample data).

The hypothesis test lets us decide whether the value of the population correlation coefficient ρ is close to zero or significantly different from zero. We decide this based on the sample correlation coefficient r and the sample size n.

If the test concludes the correlation coefficient is significantly different from zero, we say the correlation coefficient is significant.

• Conclusion: There is sufficient evidence to conclude there is a significant linear relationship between x and y because the correlation coefficient is significantly different from zero.
• What the conclusion means: There is a significant linear relationship between x and y. We can use the regression line to model the linear relationship between x and y in the population.

If the test concludes the correlation coefficient is not significantly different from zero (it is close to zero), we say the correlation coefficient is not significant.

• Conclusion: There is insufficient evidence to conclude there is a significant linear relationship between x and y because the correlation coefficient is not significantly different from zero.
• What the conclusion means: There is not a significant linear relationship between x and y. Therefore, we cannot use the regression line to model a linear relationship between x and y in the population.

Condensed — the full section is in OpenStax Statistics.

• Null hypothesis: H₀: ρ = 0.
• Alternate hypothesis: H_a: ρ ≠ 0.

• Null hypothesis H₀: The population correlation coefficient is not significantly different from zero. There is not a significant linear relationship (correlation) between x and y in the population.
• Alternate hypothesis H_a: The population correlation coefficient is significantly different from zero. There is a significant linear relationship (correlation) between x and y in the population.

There are two methods to make a conclusion. The two methods are equivalent and give the same result.

• Method 1: Use the p-value.
• Method 2: Use a table of critical values.

In this chapter, we will always use a significance level of 5 percent, α = 0.05.

Condensed — the full section is in OpenStax Statistics.

Consider the third exam/final exam example. The line of best fit is: ŷ = –173.51 + 4.83x, with r = .6631, and there are n = 11 data points. Can the regression line be used for prediction? Given a third exam score (x value), can we use the line to predict the final exam score (predicted y value)?

• H₀: ρ = 0
• H_a: ρ ≠ 0
• α = 0.05

• Use the 95 Percent Critical Values table for r with df = n – 2 = 11 – 2 = 9.
• Using the table with df = 9, we find that the critical value listed is 0.602. Therefore, the critical values are ±0.602.
• Since 0.6631 > 0.602, r is significant.
• Decision: Reject the null hypothesis.
• Conclusion: There is sufficient evidence to conclude there is a significant linear relationship between the third exam score (x) and the final exam score (y) because the correlation coefficient is significantly different from zero.

Because r is significant and the scatter plot shows a linear trend, the regression line can be used to predict final exam scores.

Testing the significance of the correlation coefficient requires that certain assumptions about the data be satisfied. The premise of this test is that the data are a sample of observed points taken from a larger population. We have not examined the entire population because it is not possible or feasible to do so. We are examining the sample to draw a conclusion about whether the linear relationship that we see between x and y in the sample data provides strong enough evidence that we can conclude there is a linear relationship between x and y in the population.

The regression line equation that we calculate from the sample data gives the best-fit line for our particular sample. We want to use this best-fit line for the sample as an estimate of the best-fit line for the population. Examining the scatter plot and testing the significance of the correlation coefficient helps us determine whether it is appropriate to do this.

• There is a linear relationship in the population that models the sample data. Our regression line from the sample is our best estimate of this line in the population.
• The y values for any particular x value are normally distributed about the line. This implies there are more y values scattered closer to the line than are scattered farther away. Assumption 1 implies that these normal distributions are centered on the line; the means of these normal distributions of y values lie on the line.
• Normal distributions of all the y values have the same shape and spread about the line.
• The residual errors are mutually independent (no pattern).
• The data are produced from a well-designed, random sample or randomized experiment.

Recall the third exam/final exam example.

We found the equation of the best-fit line for the final exam grade as a function of the grade on the third exam. We can now use the least-squares regression line for prediction.

Suppose you want to estimate, or predict, the mean final exam score of statistics students who received a 73 on the third exam. The exam scores (x values) range from 65 to 75. Since 73 is between the x values 65 and 75, substitute x = 73 into the equation. Then,

We predict that statistics students who earn a grade of 73 on the third exam will earn a grade of 179.08 on the final exam, on average.

Use the following information to answer the next two exercises. An electronics retailer used regression to find a simple model to predict sales growth in the first quarter of the new year (January through March). The model is good for 90 days, where x is the day. The model can be written as ŷ = 101.32 + 2.48x, where ŷ is in thousands of dollars.

Use the following information to answer the next three exercises. A landscaping company is hired to mow the grass for several large properties. The total area of the properties is 1,345 acres. The rate at which one person can mow is ŷ = 1350 – 1.2x, where x is the number of hours and ŷ represents the number of acres left to mow.

Use the following information to answer the next 14 exercises. contains real data for the first two decades of flu reporting.

In some data sets, there are values (observed data points) called outliers. Outliers are observed data points that are far from the least-squares line. They have large errors, where the error or residual is not very close to the best-fit line.

Outliers need to be examined closely. Sometimes, they should not be included in the analysis of the data, like if it is possible that an outlier is a result of incorrect data. Other times, an outlier may hold valuable information about the population under study and should remain included in the data. The key is to examine carefully what causes a data point to be an outlier.

Besides outliers, a sample may contain one or a few points that are called influential points. Influential points are observed data points that are far from the other observed data points in the horizontal direction. These points may have a big effect on the slope of the regression line. To begin to identify an influential point, you can remove it from the data set and determine whether the slope of the regression line is changed significantly.

You also want to examine how the correlation coefficient, r, has changed. Sometimes, it is difficult to discern a significant change in slope, so you need to look at how the strength of the linear relationship has changed. Computers and many calculators can be used to identify outliers and influential points. Regression analysis can determine if an outlier is, indeed, an influential point. The new regression will show how omitting the outlier will affect the correlation among the variables, as well as the fit of the line. A graph showing both regression lines helps determine how removing an outlier affects the fit of the model.

We could guess at outliers by looking at a graph of the scatter plot and best-fit line. However, we would like some guideline regarding how far away a point needs to be to be considered an outlier. As a rough rule of thumb, we can flag as an outlier any point that is located farther than two standard deviations above or below the best-fit line. The standard deviation used is the standard deviation of the residuals or errors.

We can do this visually in the scatter plot by drawing an extra pair of lines that are two standard deviations above and below the best-fit line. Any data points outside this extra pair of lines are flagged as potential outliers. Or, we can do this numerically by calculating each residual and comparing it with twice the standard deviation. With regard to the TI-83, 83+, or 84+ calculators, the graphical approach is easier. The graphical procedure is shown first, followed by the numerical calculations. You would generally need to use only one of these methods.

Condensed — the full section is in OpenStax Statistics.

In , the first two columns include the third exam and final exam data. The third column shows the predicted ŷ values calculated from the line of best fit: ŷ = –173.5 + 4.83x. The residuals, or errors, that were mentioned in Section 3 of this chapter have been calculated in the fourth column of the table: Observed y value – predicted y value = y – ŷ.

s is the standard deviation of all the y – ŷ = ε values, where n is the total number of data points. If each residual is calculated and squared, and the results are added, we get the SSE. The standard deviation of the residuals is calculated from the SSE as

Rather than calculate the value of s ourselves, we can find s using a computer or calculator. For this example, the calculator function LinRegTTest found s = 16.4 as the standard deviation of the residuals

• 35
• –17
• 16
• –6
• –19
• 9
• 3
• –1
• –10
• –9
• –1

We are looking for all data points for which the residual is greater than 2s = 2(16.4) = 32.8 or less than –32.8. Compare these values with the residuals in column four of the table. The only such data point is the student who had a grade of 65 on the third exam and 175 on the final exam; the residual for this student is 35.

Numerically and graphically, we have identified point (65, 175) as an outlier. Recall that recalculation of the least-squares regression line and summary statistics, following deletion of an outlier, may be used to determine whether an outlier is also an influential point. This process also allows you to compare the strength of the correlation of the variables and possible changes in the slope both before and after the omission of any outliers.

Compute a new best-fit line and correlation coefficient using the 10 remaining points.

On the TI-83, TI-83+, or TI-84+ calculators, delete the outlier from L1 and L2. Using the LinRegTTest, found under Stat and Tests, the new line of best fit and correlation coefficient are the following:

\(ŷ=-355.19+7.39x\) and \(r=0.9121\).

The slope is now 7.39, compared to the previous slope of 4.83. This seems significant, but we need to look at the change in r-values as well. The new line shows \(r=0.9121\), which indicates a stronger correlation than the original line, with \(r=0.6631\text{,}\) since \(r=0.9121\) is closer to 1. This means the new line is a better fit to the data values. The line can better predict the final exam score given the third exam score. It also means the outlier of (65, 175) was an influential point, since there is a sizeable difference in r-values. We must now decide whether to delete the outlier. If the outlier was recorded erroneously, it should certainly be deleted. Because it produces such a profound effect on the correlation, the new line of best fit allows for better prediction and an overall stronger model.

You may use Excel to graph the two least-squares regression lines and compare the slopes and fit of the lines to the data, as shown in .

You can see that the second graph shows less deviation from the line of best fit. It is clear that omission of the influential point produced a line of best fit that more closely models the data.

If you do not have the function LinRegTTest on your calculator, then you must calculate the outlier in the first example by doing the following.

First, square each |y – ŷ|.

The squares are

• 35²
• 17²
• 16²
• 6²
• 19²
• 9²
• 3²
• 1²
• 10²
• 9²
• 1².

Then, add (sum) all the |y – ŷ| squared terms using the formula

\(\underset{i=1}{\Sigma }11{(|{y}_{i}-{ŷ}_{i}|)}^{2}=\underset{i=1}{\Sigma }11{\epsilon }_{i}{}^{2}\) (Recall that y_i – ŷ_i = ε_i).

= 35² + 17² + 16² + 6² + 19² + 9² + 3² + 1² + 10² + 9² + 1²

= 2,440 = SSE.

The result, SSE, is the sum of squared errors.

Condensed — the full section is in OpenStax Statistics.

Use the following information to answer the next four exercises. The scatter plot shows the relationship between hours spent studying and exam scores. The line shown is the calculated line of best fit. The correlation coefficient is 0.69.

Use the following information to answer the next two exercises. The cost of a leading liquid laundry detergent in different sizes is given in .