Correlation and Regression: Pearson, Spearman and Least Squares

Do Two Things Move Together?

Do students who study longer score higher? Does rainfall affect maize yield? Do taller people weigh more? To answer questions like these, we collect pairs of measurements and ask two things: how strongly the two variables are related (correlation), and what equation best predicts one from the other (regression). This tutorial covers correlation and regression as taught in Further Mathematics, STA 111 and introductory research-methods courses: scatter diagrams, types of correlation, Pearson's correlation coefficient, Spearman's rank correlation, the least-squares regression line, making predictions, and the dangers of extrapolation and of confusing correlation with causation.

Scatter Diagrams

Bivariate data consists of pairs (x, y). Plotting each pair as a point gives a scatter diagram, which shows the relationship at a glance:

Positive correlation: the points rise from left to right. As x increases, y tends to increase (for example, hours of study and exam score).
Negative correlation: the points fall from left to right. As x increases, y tends to decrease (for example, the age of a car and its resale value).
No correlation: the points show no pattern (for example, shoe size and exam score).

The closer the points lie to a straight line, the stronger the correlation. A scatter diagram also shows whether the relationship is curved and whether there are outliers that might distort the calculations.

Throughout this tutorial, we use this data on six students:

Hours of study per week (x): 1, 2, 3, 4, 5, 6
Test score (y): 45, 50, 60, 62, 70, 81

Pearson's Correlation Coefficient

Pearson's product-moment correlation coefficient, r, measures the strength of a linear relationship. It always lies between −1 and +1:

r = +1: perfect positive linear correlation. All the points lie exactly on a rising line.
r = −1: perfect negative linear correlation.
r close to 0: no linear correlation.
As a rough guide, |r| above 0.7 is strong, 0.4 to 0.7 is moderate, and below 0.4 is weak.

The formula

First calculate the three summary quantities:

Sxy = Σxy − (Σx)(Σy) ÷ n
Sxx = Σx² − (Σx)² ÷ n
Syy = Σy² − (Σy)² ÷ n

Then r = Sxy ÷ √(Sxx × Syy).

Worked example 1

Calculate r for the study-hours data.

The sums are n = 6, Σx = 21, Σy = 368, Σx² = 91, Σy² = 23,430 and Σxy = 1409.

Sxy = 1409 − (21 × 368) ÷ 6 = 1409 − 1288 = 121
Sxx = 91 − 21² ÷ 6 = 91 − 73.5 = 17.5
Syy = 23,430 − 368² ÷ 6 = 23,430 − 22,570.67 = 859.33

r = 121 ÷ √(17.5 × 859.33) = 121 ÷ 122.63 ≈ 0.987.

This is a very strong positive linear correlation: more study hours go with higher scores.

Spearman's Rank Correlation Coefficient

Sometimes the data are rankings (such as the positions given by two judges), or the relationship is consistent but not linear. Then Spearman's rank correlation coefficient is used:

rₛ = 1 − 6Σd² ÷ (n(n² − 1))

Here d is the difference between the two ranks of each item and n is the number of items. Like r, it lies between −1 and +1.

Worked example 2

Two judges rank eight entries in a science fair. Judge A ranks them 1, 2, 3, 4, 5, 6, 7, 8. Judge B ranks the same entries 3, 1, 2, 5, 4, 8, 6, 7. Find rₛ.

The differences are d = −2, 1, 1, −1, 1, −2, 1, 1, so Σd² = 4 + 1 + 1 + 1 + 1 + 4 + 1 + 1 = 14.

rₛ = 1 − 6 × 14 ÷ (8 × 63) = 1 − 84 ÷ 504 ≈ 0.833.

The judges agree strongly, though not perfectly.

When raw data are given instead of ranks, rank each variable first (1 for the largest or smallest, consistently). If two values tie, give each the average of the ranks they would have occupied.

The Regression Line

Correlation measures how strongly two variables are related. Regression finds the equation of the best-fitting straight line, so that y can be predicted from x:

y = a + bx

The least-squares line is the one that makes the sum of the squared vertical distances from the points to the line as small as possible. Its coefficients are:

Gradient: b = Sxy ÷ Sxx
Intercept: a = ȳ − b x̄, where x̄ and ȳ are the means

The line always passes through the mean point (x̄, ȳ).

Worked example 3

Find the regression line of y on x for the study-hours data.

b = 121 ÷ 17.5 ≈ 6.914. The means are x̄ = 21 ÷ 6 = 3.5 and ȳ = 368 ÷ 6 ≈ 61.33.

a = 61.33 − 6.914 × 3.5 ≈ 37.13.

The regression line is y ≈ 37.13 + 6.91x.

Interpretation: each extra hour of study per week is associated with about 6.9 more marks, and a student who studies 0 hours is predicted to score about 37.

Making Predictions

Worked example 4

Use the regression line to predict the score of a student who studies (a) 4.5 hours and (b) 20 hours a week.

(a) y ≈ 37.13 + 6.914 × 4.5 ≈ 68.2. This is interpolation (predicting within the range of the data, here 1 to 6 hours), so it is reasonably reliable.

(b) y ≈ 37.13 + 6.914 × 20 ≈ 175. That is impossible on a test marked out of 100. This is extrapolation, predicting far outside the data, and it is unreliable: the straight-line pattern cannot continue forever.

Which line to use

The regression line of y on x is for predicting y from x. Predicting x from y needs the regression line of x on y, with gradient Sxy ÷ Syy, which is a different line unless the correlation is perfect.

Correlation Is Not Causation

A strong correlation does not prove that one variable causes the other. For example:

Ice-cream sales and drowning incidents are positively correlated, but ice cream does not cause drowning. Both increase in hot weather. A hidden third variable like this is called a confounding (or lurking) variable.
Two quantities that both grow over time, such as the number of mobile phones and the number of universities in Nigeria, will be correlated without one causing the other.

Establishing cause and effect needs controlled experiments, not just correlation.

Other cautions

r measures only linear relationships. Data following a perfect U-shaped curve can have r close to 0.
A single outlier can greatly change r and the regression line. Always draw the scatter diagram first.
Results from a small sample may not reflect the whole population.

Common Mistakes with Correlation and Regression

Claiming causation from a correlation.
Getting a value of r outside the range from −1 to +1, which always signals an arithmetic error.
Using Σx² when (Σx)² is needed, or the reverse, in Sxx.
Extrapolating the regression line far outside the range of the data.
Using the y-on-x line to predict x. It is designed only for predicting y.

Correlation and Regression: Practice Questions

State the type of correlation you would expect between (a) the temperature and the sales of cold drinks (b) the speed of a car and the time it takes to cover 100 km.
For the data x = 2, 4, 6, 8, 10 and y = 3, 7, 8, 12, 15, the summary quantities are Sxy = 58, Sxx = 40 and Syy = 86. Find r.
Using the data in question 2, find the regression line of y on x.
Use your line to predict y when x = 7.
Two judges rank five singers. Judge A's ranks are 1, 2, 3, 4, 5, and Judge B's are 2, 1, 4, 3, 5. Find rₛ.
A study finds a strong positive correlation between the number of fire engines at a fire and the damage caused. Does this mean fire engines cause damage? Explain.

Answers: 1. (a) Positive. (b) Negative. 2. r = 58 ÷ √(40 × 86) ≈ 0.989. 3. b = 58 ÷ 40 = 1.45, x̄ = 6 and ȳ = 9, so a = 9 − 1.45 × 6 = 0.3, and y = 0.3 + 1.45x. 4. y = 0.3 + 1.45 × 7 ≈ 10.45. 5. Σd² = 4, so rₛ = 1 − 24 ÷ 120 = 0.8. 6. No. Larger fires cause both more damage and the sending of more fire engines. The size of the fire is a confounding variable.

Conclusion

Correlation and regression describe how two variables are related. A scatter diagram shows the pattern. Pearson's r = Sxy/√(SxxSyy) measures the strength of a linear relationship, from −1 to +1, and Spearman's rₛ = 1 − 6Σd²/(n(n² − 1)) does the same for ranks. The least-squares line y = a + bx, with b = Sxy/Sxx and a = ȳ − bx̄, predicts y from x, reliably within the range of the data but not far beyond it. Above all, remember that correlation does not prove causation.