Quant from Scratch by QuantWright

Introduction to Quant Finance · Lesson 22

Regression

Fit a straight line to data by making its squared misses as small as possible. Its slope is beta, its R² says how much of the spread it removes, and on a short sample or with many inputs it can fit pure noise.

Know the basics? Read the lesson notes

Five days, one line

A day's return is what an asset gained or lost that day, in percent. Over five made-up days, a market returned −1%, 0%, +1%, +2% and +3%, and one stock returned −1%, −1%, +1%, +3% and +3%. The scatter plot below draws each day as a dot: across for the market, up for the stock.

A straight line through the dots is a guess at the stock's return from the market's. A day's miss is the stock's return minus the line's value: how far the dot sits above the line (below gives a negative miss). Each square has a miss as its side, so its area is that miss squared. Drag the line's two ends and make the total area as small as you can.

Five made-up days: the market and one stock

Drag the two round ends up or down (or select one and use the arrow keys). Colour: the squared misses.

Your line–
Sum of squared misses–

Close. No straight line gets below 1.6: the dashed line, which rises 1.2% for each 1% of the market and sits at −0.2% where the market is 0%. The line that makes the sum of squared misses as small as possible is the least-squares line, and fitting it is called linear regression.

Why square the misses?

A flat line at the stock's average, +1%, misses the five days by −2, −2, 0, +2 and +2. The least-squares line misses by +0.4, −0.8, 0, +0.8 and −0.4. Add up each set of misses as they are, without squaring. Which total is smaller?

Both add up to 0: −2 − 2 + 0 + 2 + 2 = 0, and 0.4 − 0.8 + 0 + 0.8 − 0.4 = 0.
Both add up to 0: −2 − 2 + 0 + 2 + 2 = 0, and 0.4 − 0.8 + 0 + 0.8 − 0.4 = 0.
−2 − 2 + 0 + 2 + 2 = 0, and 0.4 − 0.8 + 0 + 0.8 − 0.4 = 0.

Misses above the line and below it cancel, so a plain sum can't tell a good line from a bad one. Squaring makes every miss count, and a miss twice as big counts four times as much, as in a standard deviation (how far numbers typically land from their average, built from squared distances). Squared, the flat line's misses add up to 4 + 4 + 0 + 4 + 4 = 16, the least-squares line's to 0.16 + 0.64 + 0 + 0.64 + 0.16 = 1.6.

Theory

Slope and intercept from scratch

You don't have to drag: the least-squares line has a formula. Its slope is how far the line rises for each 1 that x moves; its intercept is its value where x is 0. The slope is the covariance of x and y (their distances from their averages multiplied pair by pair, added up and divided by n − 1) divided by the variance of x (its squared distances from its average, added up and divided by n − 1). The n − 1 is a small-sample correction, and here it cancels.

b = cov(x, y) ÷ var(x)
a = ȳ − b × x̄
x, y
the paired numbers, n pairs (here the market's and the stock's returns)
x̄, ȳ
their averages
cov(x, y)
the sum of (x − x̄)(y − ȳ), divided by n − 1
var(x)
the sum of (x − x̄)², divided by n − 1: the square of x's standard deviation
b, a
the slope and the intercept of the least-squares line
Worked example: your five days

Averages: market (−1 + 0 + 1 + 2 + 3) ÷ 5 = 1, stock (−1 − 1 + 1 + 3 + 3) ÷ 5 = 1. Distances: market −2, −1, 0, +1, +2; stock −2, −2, 0, +2, +2. Products: 4, 2, 0, 2, 4, adding up to 12, so cov = 12 ÷ 4 = 3. Squared market distances: 4, 1, 0, 1, 4, adding up to 10, so var = 10 ÷ 4 = 2.5.

b = 3 ÷ 2.5 = 1.2, and a = 1 − 1.2 × 1 = −0.2: the line you found by dragging. Both divide by n − 1 = 4, so it cancels: b = 12 ÷ 10.

Holds when
  • you want the straight line that misses y least, counting each miss squared;
  • every day counts the same.
Breaks when

The link is curved (the Correlation and diversification lesson showed a perfect U-shaped link with a correlation near 0). One extreme day pulls the line a lot, because its miss is squared.

New numbers. x is 1, 2 and 3; y is 2, 3 and 7. What is the slope?

Averages 2 and 4. Distances: x −1, 0, +1; y −2, −1, +3. Products 2, 0, 3, adding up to 5; squared x distances 1, 0, 1, adding up to 2. b = 5 ÷ 2 = 2.5 (the n − 1 cancels), and a = 4 − 2.5 × 2 = −1.

Beta: the slope on the market

When x is a market's return and y a stock's, the slope has its own name, beta: how far the stock tends to move for each 1% the market moves. Below are 250 simulated days of one market and two stocks, each with its least-squares line. Their correlation with the market (from −1 to +1: how closely the dots follow a straight line) is 0.48 for stock A and 0.89 for stock B. Which stock has the larger beta?

One market, two stocks, 250 days (simulated)

One dot a day: across for the market, up for the stock. The two charts share one scale. Colour: the least-squares line.

Stock A · correlation 0.48

Stock B · correlation 0.89

Stock A's line is steeper: beta 1.49 against 0.50. Its dots scatter widely around that line, which is why its correlation is lower.
Stock B's dots sit closer to their line, which is what its higher correlation says, but its line is flatter: beta 0.50 against stock A's 1.49.
Stock A's line is three times as steep: beta 1.49 against 0.50. The correlations differ for another reason: how closely the dots follow each line.

Correlation says how closely the dots follow the line; beta says how steep it is. The two are linked. The correlation ρ ("rho") is cov ÷ (sx × sy), where sx and sy are the market's and the stock's standard deviations, and var(x) is sx², so beta = cov ÷ var = ρ × sy ÷ sx. Stock A was built with a beta of 1.5 and stock B with 0.5.

Theory

R²: the share of the spread the line removes

Before any line, the stock's spread is the sum of its squared distances from its own average: the flat line's 16. After the least-squares line, 1.6 is left. The share the line removed is R² ("R squared"). Textbooks say the line "explains" that share; it means only that the misses shrink on these days, not that x causes y.

R² = 1 − [sum of (y − ŷ)²] ÷ [sum of (y − ȳ)²]
ŷ
"y-hat", the line's value, a + b × x; so y − ŷ are its misses
y − ȳ
the distances from y's average: the flat line's misses
R²
for a least-squares line with an intercept, on the data it was fitted to: from 0 (no better than the average) to 1 (every dot on the line); with one input it equals the correlation squared
Worked example: your five days

1 − 1.6 ÷ 16 = 1 − 0.1 = 0.9: on these five days the line removes 90% of the stock's spread. The correlation is 12 ÷ √(10 × 16) ≈ 0.95, and squared that is 144 ÷ 160 = 0.9.

Holds when
  • the line is fitted by least squares with an intercept;
  • you read R² for the data the line was fitted to.
Breaks when

R² says nothing about cause, and on new data it is usually lower. An input added never lowers it on the data used, even an input of pure noise (two cards on).

New numbers. Over 100 days a stock's squared distances from its average add up to 40. Fitted on the market's returns, its squared misses add up to 30. What is R²?

1 − 30 ÷ 40 = 1 − 0.75 = 0.25. The line removes a quarter of the spread, and the correlation is √0.25 = 0.5 (or −0.5 if the line falls).

A line through noise

Every line has a slope, even with nothing there. Take a made-up signal X and a stock's return Y with nothing between them: each day of each is its own bell-curve draw (most days near the average, fewer far out) with a standard deviation of 1%, so the true slope is 0. Call a slope "big" at 0.3 or more either way: the stock moving 0.3% for each 1% of the signal. Guess first: of 1,000 samples of 20 days, how many show a big slope?

1,000 samples of each length, true slope 0 (simulated)

Each column counts the samples whose measured slope landed in a range 0.05 wide. Colour: 0.3 or more either way.

Your guess:
Days in a sample:
The middle 950 of 1,000–
0.3 or more either way–

In this simulation, 203 of the 1,000 samples of 20 days showed a slope of 0.3 or more either way, with nothing there: about one in five. Of 1,000 samples of 500 days, 0 did. On a short sample the slope is mostly luck.

More inputs, a better fit, on the days used

A regression can take several inputs at once, each with its own slope; least squares picks them all together. Here a stock's return over 20 days is pure noise, and so are 19 made-up inputs: none says anything about the stock. The line is fitted on those 20 days, then tried on 20 new days. Guess first: as inputs are added, what does R² on the 20 days used do?

One sample: 20 days used, then 20 new days (simulated)

Dots: the stock's return each day. Colour: the fitted line's value. Shaded: the 20 days used for fitting.

Your guess:
Inputs:

R² on the days used, this sample–
Average R² on the days used, 1,000 samples–
Missed the new days by more than the flat line–

An added input can only shrink the squared misses on the days used, because least squares could always give it a slope of 0. In this simulation R² on the days used never fell with an input added, in 1,000 of 1,000 samples, and reached 1.00 at 19 inputs: 20 numbers fitted to 20 days pass through every dot. On the 20 new days, the 10-input line missed by more than a flat line at the 20 days' average (no inputs at all) in 952 of 1,000. Fitting noise in the data used is called overfitting; the Overfitting lesson in Backtesting Essentials shows it in a backtest.

A hidden driver

Two more simulated series, X and Y, over 250 days. Each day a hidden series Z moves both: X is Z plus a move of its own, and Y is Z plus another, all three separate bell-curve draws of one size. X has no part in Y. Yet fit Y on X and the slope is 0.49 on these days, and 0.50 on average over 1,000 such samples.

X and Y, 250 days (simulated)

X's slope, these 250 days–

Give the regression Z as a second input, so X's slope is measured with Z's part counted separately. What will X's slope be?

Z was all that X and Y shared. With Z in the regression, nothing is left for X to account for, so its slope falls to about 0.
Z was all that X and Y shared. With Z in the regression, nothing is left for X to account for, so its slope falls to about 0, not up.
Z was all that X and Y shared. With Z in the regression, nothing is left for X to account for.

On these days it was −0.03. Across 1,000 samples it averaged 0.00 and was 0.3 or more either way in 0, against 1,000 for the slope of Y on X alone. The figure's second view shows X and Y with Z's part removed from each.

A slope, like a correlation, can't tell X moving Y from a third series moving both, a confounder: correlation is not causation. Adding Z fixed it here only because Z was known and measured; nothing in X and Y alone shows that Z exists.

The slope is an estimate

Stock A, on the beta card, was built with a true beta of 1.5, and its 250 days measured 1.49. Now build 1,000 more samples of 250 days the same way and measure beta in each. Where do the middle 950 land?

Much wider: from 1.17 to 1.81. Each sample's own moves of the stock tilt its line a little.
They ran from 1.17 to 1.81. Each sample's own moves of the stock tilt its line a little.
Narrower than that: from 1.17 to 1.81. With 250 days, much of the luck cancels.

Beta measured on 1,000 samples of 250 days, true beta 1.5 (simulated)

Each column counts the samples whose measured beta landed in a range 0.05 wide. Shaded: the middle 950, from 1.17 to 1.81.

A measured slope carries its own uncertainty. Its usual measure is the slope's standard error: the typical gap between a measured slope and the true one, about 0.16 here. And a beta measured on past days describes those days; these simulated stocks keep one beta throughout, so they can't show it changing.

For the preciseThe measured beta landed within two standard errors of the true 1.5 in 953 of 1,000 samples, near the 95 in 100 the formula aims at when the days are independent (one day's result doesn't change the next's) and beta stays the same. The standard error of the slope is √[(sum of (y − ŷ)²) ÷ (n − 2) ÷ (sum of (x − x̄)²)], for independent days whose misses have one size throughout. For stock A's 250 days it is 0.17. Across the 1,000 samples the measured betas averaged 1.5 and ran from 0.95 to 2.09.

What this means for your backtest

A backtest runs a trading rule over past prices to see what it would have done. Turn over three habits.

  1. Read a slope with its sample size.

    On a short sample, a signal with nothing in it often shows a slope that looks big.

Practice 1 of 2

"It moves 0.6 times the market"

The situationA friend's note: "Over the last year the stock's correlation with the market was 0.6, so it moves 0.6% for each 1% the market moves." Over the same year the stock's daily returns had a standard deviation of 3%, and the market's 1.5%.

From these numbers, what is the stock's beta?

Hint: beta = ρ × sy ÷ sx, with the stock as y and the market as x.

0.6 × 3 ÷ 1.5 = 1.2: the stock tended to move 1.2% for each 1% of the market over that year. The correlation says how closely the days follow the line, not how steep it is; the two numbers match only when the two standard deviations are equal.
Practice 2 of 2

"Our model explains 60%"

The situationA seller: "Our model has 10 inputs, and over the last 20 trading days it explains 60% of the stock's daily returns (R² = 0.6)."

What does that R² tell you?

R² is measured on the days the model was fitted to. In the lesson's simulation, a 10-input line fitted to 20 days of pure noise missed new days by more than a flat line in 952 of 1,000 samples.
In the lesson's simulation, 10 inputs of pure noise on 20 days reached an R² of 0.6 or more in 324 of 1,000 samples. Only days the model never saw can show whether it removes anything.
R² only says how much the squared misses shrank on those days; it says nothing about cause. And 10 inputs of pure noise on 20 days reached 0.6 or more in 324 of 1,000 simulated samples.
✓

Lesson done

Linear regression fits the straight line whose squared misses add up to as little as possible. Its slope is the covariance of x and y divided by the variance of x; for a stock on a market it is called beta. R² is the share of the spread the line removes, on the data it was fitted to; it says nothing about cause. In this lesson's simulation a line through 20 days of noise showed a slope of 0.3 or more either way in 203 of 1,000 samples, and inputs of pure noise never lowered R² on the data used. A measured slope describes past days and carries its own uncertainty, its standard error.

Key ideas

Least squares
fitting a line by making the sum of its squared misses as small as possible
Linear regression
fitting a least-squares line: one input, or several, each with its own slope
Slope / intercept
how far the line rises for each 1 that x moves / the line's value where x is 0
Beta
the slope of a stock's returns on a market's: how far the stock tended to move for each 1% the market moved, over the days measured
R²
the share of y's spread (squared distances from its average) the line removes, on the data it was fitted to; not a measure of cause
Overfitting
a fit that matches noise in the data used, so it looks better there than on new data
Confounder
a third series that moves two others, giving one a slope on the other though neither moves the other
Standard error of the slope
the typical distance of a measured slope from the true one, estimated from one sample when the days are independent

Formulas

b = cov(x, y) ÷ var(x) = ρ × sy ÷ sx · a = ȳ − b × x̄

R² = 1 − sum of (y − ŷ)² ÷ sum of (y − ȳ)²

Both describe the data the line was fitted to; on a short sample, or with many inputs, they can come from noise.

Lesson notes

Linear regression fits the straight line through pairs of numbers whose squared misses add up to as little as possible (least squares). Its slope is the covariance of x and y divided by the variance of x; for a stock's returns against a market's it is called beta. R² is the share of y's spread the line removes, on the data it was fitted to. A line can mislead: on a short sample or with many inputs it fits pure noise, and a hidden third series can give a slope where one series has no part in the other. A measured slope describes past days and carries its own uncertainty (its standard error, below). Every count below is from the lesson's simulation, in which a computer draws every day.

Fitting a line by least squares

A day's return is what an asset gained or lost that day, in percent. Over five made-up days a market returned −1%, 0%, +1%, +2% and +3%, and a stock returned −1%, −1%, +1%, +3% and +3%. A scatter plot draws each day as a dot: across for the market, up for the stock. A straight line through the dots is a guess at the stock's return from the market's; a day's miss is the stock's return minus the line's value. The least-squares line makes the sum of the squared misses as small as possible; fitting it is linear regression. For the five days no line gets below 1.6: the line that rises 1.2% for each 1% of the market and sits at −0.2% where the market is 0%.

The misses are squared because misses above and below a line cancel. A flat line at the stock's average, +1%, misses by −2, −2, 0, +2 and +2; the least-squares line by +0.4, −0.8, 0, +0.8 and −0.4. Both sets add up to 0, so a plain sum can't tell them apart. Squared, they add up to 16 and 1.6. Squaring makes every miss count and a miss twice as big count four times as much, as in a standard deviation (how far numbers typically land from their average, built from squared distances).

Beta

When x is a market's return and y a stock's, the slope is called beta: how far the stock tended to move for each 1% the market moved. Over 250 simulated days, stock A (built with a true beta of 1.5) measured a beta of 1.49 and a correlation with the market (from −1 to +1: how closely the dots follow a straight line) of 0.48. Stock B (built with a beta of 0.5) measured 0.50 and 0.89. The stock with the steeper line had the lower correlation: correlation says how closely the dots follow the line, beta how steep it is. They are linked: since ρ = cov ÷ (sx × sy), cov ÷ var = ρ × sy ÷ sx, where sy and sx are the stock's and the market's standard deviations.

Three traps

A short sample. A made-up signal and a stock's return, each day of each drawn separately from a bell curve (most days near the average, fewer far out) with a standard deviation of 1%, have nothing between them: a true slope of 0. Of 1,000 simulated samples of 20 days, 203 showed a slope of 0.3 or more either way; of 1,000 samples of 500 days, 0. On a short sample the slope is mostly luck.

More inputs. A regression can take several inputs, each with its own slope. With a stock's return over 20 days and 19 inputs, all pure noise, R² on the 20 days used never fell with an input added, in 1,000 of 1,000 simulated samples, and reached 1.00 at 19 inputs (20 numbers fitted to 20 days pass through every dot). An added input can only shrink the squared misses on the days used, because least squares could always give it a slope of 0. On 20 new days the 10-input line missed by more than a flat line at the days-used average in 952 of 1,000 samples. Fitting noise in the data used is overfitting.

A hidden driver. Two simulated series, X and Y, are each a hidden series Z plus a move of their own, all three separate bell-curve draws of one size; X has no part in Y. Across 1,000 samples of 250 days, the slope of Y on X averaged 0.50 and was 0.3 or more in all 1,000. With Z as a second input, X's slope averaged 0.00 and was 0.3 or more either way in 0. A third series that moves two others is a confounder: a slope, like a correlation, can't tell one from a cause. Adding Z worked only because Z was known and measured.

The slope is an estimate

Across 1,000 simulated samples of 250 days built like stock A, the measured betas' middle 950 ran from 1.17 to 1.81, around the true 1.5. The usual measure of that spread is the slope's standard error, the typical gap between a measured slope and the true one, which a textbook formula estimates from one sample: √[(sum of (y − ŷ)²) ÷ (n − 2) ÷ (sum of (x − x̄)²)]. It averaged 0.16, and the measured beta landed within two standard errors of 1.5 in 953 of 1,000 samples, near the 95 in 100 the formula aims at when the days are independent (one day's result doesn't change the next's) and beta stays the same. A beta measured on past days describes those days; the simulated stocks keep one beta throughout, so they can't show it changing.

Formula: slope and intercept

The least-squares line through n pairs, counting each miss squared and every pair the same:

b = cov(x, y) ÷ var(x)
a = ȳ − b × x̄
x̄, ȳ
the averages of x and y
cov(x, y)
the sum of (x − x̄)(y − ȳ), divided by n − 1
var(x)
the sum of (x − x̄)², divided by n − 1
b, a
the slope and the intercept
Worked example: the five days

Averages 1 and 1. Distances: market −2, −1, 0, +1, +2; stock −2, −2, 0, +2, +2. Products 4, 2, 0, 2, 4, adding up to 12, so cov = 12 ÷ 4 = 3. Squared market distances 4, 1, 0, 1, 4, adding up to 10, so var = 10 ÷ 4 = 2.5. b = 3 ÷ 2.5 = 1.2 and a = 1 − 1.2 × 1 = −0.2. With 1, 2, 3 against 2, 3, 7: b = 5 ÷ 2 = 2.5 and a = 4 − 2.5 × 2 = −1.

Holds whenBreaks when

The link is curved. One extreme day pulls the line a lot, because its miss is squared. Short samples, many inputs and hidden drivers can each produce a slope with nothing behind it.

Formula: R²

The share of y's spread a least-squares line with an intercept removes, on the data it was fitted to:

R² = 1 − [sum of (y − ŷ)²] ÷ [sum of (y − ȳ)²]
ŷ
the line's value, a + b × x; so y − ŷ are its misses
y − ȳ
the distances from y's average
R²
from 0 to 1 on the data fitted; with one input, the correlation squared
Worked example: the five days

1 − 1.6 ÷ 16 = 0.9; the correlation is 12 ÷ √(10 × 16) ≈ 0.95, and squared that is 144 ÷ 160 = 0.9. Squared distances adding up to 40 and squared misses adding up to 30 give 1 − 30 ÷ 40 = 0.25.

Holds whenBreaks when

R² says nothing about cause, and on new data it is usually lower. An input added never lowers it on the data used, even an input of pure noise.

One more worked example

Made-up numbers

Three days: x is 0, 2 and 4; y is 1, 2 and 6. Averages 2 and 3. Distances: x −2, 0, +2; y −2, −1, +3. Products 4, 0, 6, adding up to 10; squared x distances 4, 0, 4, adding up to 8. b = 10 ÷ 8 = 1.25 and a = 3 − 1.25 × 2 = 0.5. The line's values are 0.5, 3 and 5.5, so the misses are +0.5, −1 and +0.5, and their squares add up to 0.25 + 1 + 0.25 = 1.5. The squared distances of y from its average are 4, 1 and 9, adding up to 14. R² = 1 − 1.5 ÷ 14 ≈ 0.89, on three days, which is far too few to say anything beyond them.

Common mistakes

Check yourself

  1. Over four days x is 1, 2, 3 and 4, and y is 3, 5, 4 and 8. What are the slope and the intercept of the least-squares line?
    Answer

    Averages 2.5 and 5. Distances: x −1.5, −0.5, +0.5, +1.5; y −2, 0, −1, +3. Products 3, 0, −0.5, 4.5, adding up to 7; squared x distances 2.25, 0.25, 0.25, 2.25, adding up to 5. b = 7 ÷ 5 = 1.4, and a = 5 − 1.4 × 2.5 = 1.5.

  2. A stock's covariance with the market is 2 (in percent times percent) and the market's variance is 1.6. What is the stock's beta?
    Answer

    2 ÷ 1.6 = 1.25: over those days the stock tended to move 1.25% for each 1% of the market.

  3. Two series have a correlation of −0.8. What R² does a least-squares line of one on the other have?
    Answer

    With one input, R² is the correlation squared: (−0.8)² = 0.64. The line falls, but it still removes 64% of the spread on the data fitted.

  4. On 20 days, a signal and a stock's return give a slope of 0.35. What does the lesson's simulation say about that?
    Answer

    In its simulation, a signal with nothing in it showed a slope of 0.3 or more either way in 203 of 1,000 samples of 20 days, so 0.35 on 20 days is well within what luck gives. On 500 days, 0 of 1,000 did.

  5. Adding a fifth input raised a model's R² on its data from 0.30 to 0.34. Does that show the input matters?
    Answer

    No. An added input can never lower R² on the data used, because least squares could give it a slope of 0; in the lesson's simulation R² never fell with an input of pure noise added, in all 1,000 samples. Only data the model never saw can show whether the input matters.

Try it yourself

Run the script. Every number in this lesson comes from one Python script (below), which uses its own seeded draws, not the course's shared simulated market. It builds the series, fits every line and counts every result. Change the sample length, the number of inputs or the size of the hidden series, and see what moves.

By hand, in a spreadsheet. Put x in column A and y in column B, one day a row. =SLOPE(B:B,A:A) gives the slope, =INTERCEPT(B:B,A:A) the intercept and =RSQ(B:B,A:A) R². =COVARIANCE.S(A:A,B:B)/VAR.S(A:A) gives the slope again, from the formula.

The code behind every number (Python, numpy; seed 2026112200)
Loading…

Honest notes

Where this comes from