Quant from Scratch by QuantWright

Backtesting Essentials · Lesson 3

Survivorship bias

Did the test only see the winners? Why testing on today's list of stocks leaves out the ones that died, and how much that flatters a result.

Know the basics? Read the lesson notes

A backtest on today's list

A stock is a small share of a company, bought and sold on an exchange; its price moves every trading day. A backtest replays a plan over past prices to see what it would have made.

The plan: put $100 into every stock on a list at the start of 2015, and hold to the end of 2024. No trading costs. For the list, you take the stocks trading today (here, the end of 2024): 311 of them, each with prices going back to 2015. These are simulated stocks: prices made up by a program, not taken from any real market.

Today's 311 stocks, bought in 2015 and held (simulated)

The line: what $100 in the average stock on the list was worth on each day. Dotted line: the $100 put in.

Date–
$100 became, on average$100.00

On average, $100 became $284.71: a gain of $184.71 on each $100, or +184.7% over ten years.

Who is missing?

Today's list holds only the stocks still trading today. In 2015 there were more. Some stopped trading along the way: they were delisted, taken off the exchange. In real markets that happens when a company goes bust, its price falls too far, or another company buys it. In this simulation one rule decides it: a stock is delisted on the first day its price closes (its price at the end of the day) below a quarter of its 2015 price, so below $25 for each $100 put in. A holder gets that last price, and the stock is gone.

Press Play to watch all the stocks that traded in 2015, over the ten years.

Every stock trading in January 2015, ten years (simulated)

One line per stock: what its $100 was worth. The scale is stretched at the bottom, so low values stay readable: from $100 to $1,000 takes as much room as from $10 to $100 would. Dashed line: $25. A line that ends with an orange mark is a stock delisted there; it then fades.

Date–
Still trading–
Delisted–

500 stocks traded in 2015. 189 were delisted before the end of 2024. The 311 left are today's list. The 189 left no row on it, so the backtest you just ran never saw them.

Put them back: guess first

On today's list, $100 became $284.71 on average. Now run the same plan on all 500 stocks that traded in 2015. The 189 delisted ones count at the last price a holder got.

What will the average $100 become?

Keep your guess in mind. On the next screen you put the 189 back.

All 500, against today's 311

The same plan, the same ten years. The only change is which list you test. Switch between the two.

Buy and hold: today's list against every stock that existed (simulated)

Coloured: the list you picked. Grey: the other one. Dotted line: the $100 put in.

List
Stocks–
$100 became, on average–
Return–

A percentage point is the gap between two percentages: from +86.3% to +184.7% is 184.7 − 86.3 = 98.4 percentage points. Today's list more than doubled the gain. This is survivorship bias: a test that only sees what survived to the end. (A bias is an error that leans one way, rather than one that comes from chance.)

Why it only goes one way

Here is where each of the 500 stocks ended. The bars count stocks by what their $100 became. The triangle marks the average. Take the delisted ones out and watch the average.

What $100 became in each of the 500 stocks (simulated)

Bar height: how many stocks ended in that range. Orange: delisted (dashed once taken out). The bottom scale is stretched like the chart of every stock earlier: from $100 to $1,000 takes as much room as from $10 to $100. Each bar covers values about a quarter higher than the one before.

Stocks500
Average$186.34

All 189 delisted stocks ended below $25, far below the average of all 500, $186.34. Take numbers that sit below an average out of it, and the average rises (the low numbers that were pulling it down are gone). Today's list didn't drop stocks at random: it dropped them for doing badly. So here it moves the result up. Later you'll see the rare cases where it doesn't.

It is a peek, like the one in the last lesson. Choosing the list at the end uses something nobody knew in 2015: which stocks would survive.

Not only stocks

Any list chosen by how things ended has the same bias, whether it lists stocks, funds or strategies. A fund pools many people's money and invests it; a strategy is a fixed set of trading rules. For each list, say how it was chosen.

Every stock that traded on the exchange on the first day of 2015, each kept until it stopped trading.

At the start. The list is fixed in 2015, before anyone knew which would last. The stocks that died stay in it, up to their last price.

Not quite. The list is fixed in 2015, before anyone knew which would last. The stocks that died stay in it.

A seller's page of the strategies they still run, each with its results since it began.

Not quite. Strategies that lost were likely stopped and taken off the page. You see the ones that are still worth showing.

By how it ended. Strategies that lost were likely stopped and taken off the page. You see the ones that are still worth showing.

Sorted0 of 2
Theory

Survivorship bias, measured

Survivorship bias is the gap between a result measured on the survivors only and the same result on everything that existed at the start. For a buy-and-hold test, the result is the average return of the stocks on the list:

bias = R̄S − R̄all = d × (R̄S − R̄D)
R̄S
the average return of the survivors (the stocks still trading at the end)
R̄all
the average return of everything that existed at the start
R̄D
the average return of the dead ones (delisted), up to their last price
d
the share that died: the number delisted ÷ the number at the start

R̄ is an average of returns, with each stock counting the same because each got $100; a return here is the total gain over the ten years, in %. Why the two forms agree: the average of all is the survivors' and the dead ones' averages, each weighted by its share, R̄all = (1 − d) × R̄S + d × R̄D. Subtracting that from R̄S: R̄S − (1 − d) × R̄S − d × R̄D = d × (R̄S − R̄D), the second form. It says the bias grows with how many died and how far they fell behind the survivors.

Worked example: the 500 stocks

Survivors +184.7%, all 500 +86.3%: bias = 184.7 − 86.3 = 98.4 points. The 189 delisted ones averaged −75.5% ($100 became $24.46). Second form: d = 189 ÷ 500 = 0.378, and 0.378 × (184.7 − (−75.5)) = 0.378 × 260.2 = 98.4. The same answer.

Holds when
  • the plan is buy-and-hold (it gains when the stocks rise);
  • the dead ones died for doing badly, so R̄D is below R̄S. Then the bias is positive: the survivors flatter the result.
Breaks when

Some leave for good reasons, such as another company buying them at a higher price: then the bias is smaller. A rule that bets on prices falling would have made its best trades on the dead ones, so for it the bias flips and the survivors make it look worse.

Try it on new numbers. 100 funds started ten years ago; 20 were closed. The 80 still open returned +60% on average, the 20 closed ones −30%. How many percentage points does a table of today's funds flatter the average?

d = 20 ÷ 100 = 0.2. Bias = 0.2 × (60 − (−30)) = 0.2 × 90 = 18 points. Check: all 100 averaged (1 − 0.2) × 60 + 0.2 × (−30) = 48 − 6 = 42%, and 60 − 42 = 18.
d = 20 ÷ 100 = 0.2. Bias = 0.2 × (60 − (−30)) = 0.2 × 90 = 18 points. Check: all 100 averaged (1 − 0.2) × 60 + 0.2 × (−30) = 48 − 6 = 42%, and 60 − 42 = 18. Careful with the minus sign: 60 − (−30) is 90.
Theory

The fix: the list as it stood

Test on the list as it stood at the start, not as it stands today: every stock that was trading then, the dead ones included up to their last price. This is a point-in-time universe (the universe is the list of stocks a test may choose from). Its average is every stock's return, dead or alive, added up and divided by the number that existed:

R̄all = (NS × R̄S + ND × R̄D) ÷ (NS + ND)
NS, ND
how many survived, how many died
R̄S, R̄D
their average returns, the dead ones' up to their last price
R̄all
the honest average: what the plan made on the list as it stood at the start
Worked example: the 500 stocks

(311 × 184.7 + 189 × (−75.5)) ÷ 500 = (57,441.7 − 14,269.5) ÷ 500 = 43,172.2 ÷ 500 = +86.3%, the result you saw with all 500 switched on.

Holds when
  • you have the list as it was on the start date, not rebuilt later;
  • you have the dead ones' prices up to the last one a holder could get;
  • for a test that picks stocks again later, you use the list as it stood on each of those dates.
Breaks when

The data has quietly lost the dead ones' last prices or rows (lesson B4, Data quality). Or the list was picked after seeing results, such as keeping the settings that tested best (lesson B7, The best of many).

Try it on new numbers. 40 stocks traded at the start. The 30 survivors returned +50% on average; the 10 delisted ones −80%. What is the honest average return of all 40, in %?

(30 × 50 + 10 × (−80)) ÷ 40 = (1,500 − 800) ÷ 40 = 700 ÷ 40 = +17.5%. Today's list would have shown +50%.
(30 × 50 + 10 × (−80)) ÷ 40 = (1,500 − 800) ÷ 40 = 700 ÷ 40 = +17.5%. Divide by all 40, not by the 30 survivors.

What this means for your backtest

Three habits keep the dead in the test. Turn them over one at a time.

  1. Ask when the list was made.

    A list made at the end chose its members by how they ended. A free download of "all stocks" can be today's stocks only.

Practice 1 of 3

A fund table

The situationFunds that do badly are often closed or merged into others. A website says it shows every fund in one category, each with its ten-year record. The average is +9% a year. The small print: "Funds currently available."

What do you make of the +9%?

"Currently available" means the funds still open. Funds that were closed, often for doing badly, dropped out, so the average is a survivors' average.
It lists every fund in the category today. Funds closed during the ten years aren't there, and they were often the weak ones: a survivors' average.
A fund without ten years of history can't have a ten-year record, so new funds don't drag this average. What's missing are the funds that closed, often the weak ones.
The honest number needs every fund that was open ten years ago, closed ones included.
Practice 2 of 3

A friend's backtest

The situationA friend bought-and-held today's list of 80 stocks over ten years: +60% on average. You find 20 more stocks that were trading at the start and were later delisted. Up to their last prices, they returned −90% on average.

By how many percentage points did today's list flatter your friend's result?

Hint: 100 stocks existed at the start, and 20 of them died. Use bias = d × (R̄S − R̄D), or work out the average of all 100 first.

d = 20 ÷ 100 = 0.2, and 0.2 × (60 − (−90)) = 0.2 × 150 = 30 points. All 100 averaged (80 × 60 + 20 × (−90)) ÷ 100 = +30%, half what today's list showed.
d = 20 ÷ 100 = 0.2, and 0.2 × (60 − (−90)) = 0.2 × 150 = 30 points. All 100 averaged (80 × 60 + 20 × (−90)) ÷ 100 = +30%, and 60 − 30 = 30.
Practice 3 of 3

Three strategies, all up

The situationA seller's page shows three strategies, each up more than 40% since it began. An old copy of the page, from two years ago, listed twelve.

What can you say about the seller's strategies as a whole?

The three shown are the survivors of twelve. The nine dropped ones are missing from the average, and they were likely dropped for doing badly.
Three of twelve can do well by luck alone, even if none has any real advantage (lesson B7, The best of many, shows how often). So this doesn't show a quarter "work", and first you need the other nine.
The three shown are the survivors of twelve. Without the other nine, you can't work out the average of all twelve, which is the honest number.
✓

Lesson done

A test on today's list sees only what survived. Stocks dropped for doing badly are missing, so the average comes out too high: on 500 simulated stocks, today's list showed +184.7% where all that existed made +86.3%. Test on the list as it stood at the start, dead ones included, and ask the same question of any list of winners you're shown.

Key ideas

Delisted
taken off the exchange: the stock stops trading
Survivorship bias
the gap between a result on the survivors only and on everything that existed at the start
Percentage point
the gap between two percentages: +86.3% to +184.7% is 98.4 points
Universe
the list of stocks a test may choose from
Point-in-time universe
the list as it stood on each date, including stocks that later died

Formulas

bias = R̄S − R̄all = d × (R̄S − R̄D)  ·  R̄all = (NS × R̄S + ND × R̄D) ÷ (NS + ND)

Lesson notes

Survivorship bias is the gap between a result measured only on what survived to the end and the same result on everything that existed at the start. A list of today's stocks leaves out the ones that stopped trading, usually because they did badly, so a test on it comes out too high. The fix is to test on the list as it stood at the start.

A backtest on today's list

A stock is a small share of a company, bought and sold on an exchange. A backtest replays a plan over past prices to see what it would have made. The plan: put $100 into every stock on a list at the start of 2015 and hold to the end of 2024, with no trading costs. Buying once and keeping to the end is buy-and-hold. The list is the stocks trading today, at the end of 2024: 311 simulated stocks (prices made up by a program), each with prices back to 2015. On average, $100 became $284.71, a gain of $184.71 on each $100, or +184.7% over ten years.

Who is missing?

Today's list holds only the stocks still trading. Some stopped along the way: they were delisted, taken off the exchange. In real markets that happens when a company goes bust, its price falls too far, or another company buys it. The simulation uses one rule: a stock is delisted on the first day its price closes (its price at the end of the day) below a quarter of its 2015 price, so below $25 for each $100 put in. A holder gets that last price. In the lesson's simulation, 500 stocks traded in 2015 and 189 were delisted, 189 ÷ 500 = 37.8%. The 311 left are today's list, so the backtest never saw the 189.

Put them back

Run the same plan on all 500, with the delisted ones counting at their last price. $100 became $186.34 on average, +86.3%, not $284.71. A percentage point is the gap between two percentages: +184.7% against +86.3% is 184.7 − 86.3 = 98.4 percentage points. Today's list more than doubled the gain. This is survivorship bias. (A bias is an error that leans one way, rather than one that comes from chance.)

Why it usually pushes the result up

All 189 delisted stocks ended below $25, far below the average of all 500, $186.34. The delisted ones averaged $24.46, which is −75.5%. Take numbers that sit below an average out of it, and the average rises. Today's list did not drop stocks at random. It dropped them for doing badly, so it moves the result up. It is a peek, like the last lesson's: choosing the list at the end uses something nobody knew in 2015, which stocks would survive. This is one simulated rule in which every death is a bad one. Real delistings include takeovers, often at a higher price, so the real bias can be smaller.

Not only stocks

Any list chosen by how things ended has the same bias, whether it lists stocks, funds or strategies. A fund pools many people's money and invests it; a strategy is a fixed set of trading rules. Every stock on the exchange on the first day of 2015, each kept until it stopped trading, is a list chosen at the start. A seller's page of the strategies they still run is chosen by how they ended, because strategies that lost were likely stopped and taken off the page. So ask when the list was made. A free download of "all stocks" can be today's stocks only. And before you believe the average of any list of winners, ask how many started and what became of the rest.

The list as it stood

Test on the list as it stood at the start: every stock trading then, the dead ones included up to their last price. This is a point-in-time universe (the universe is the list of stocks a test may choose from). A test that picks stocks again later must use the list as it stood on each of those dates.

Formula: survivorship bias

For a buy-and-hold test, the result is the average return of the stocks on the list. A return here is the total gain over the ten years, in %, and each stock counts the same because each got $100.

bias = R̄S − R̄all = d × (R̄S − R̄D)
R̄S
the average return of the survivors (the stocks still trading at the end)
R̄all
the average return of everything that existed at the start
R̄D
the average return of the dead ones (delisted), up to their last price
d
the share that died: the number delisted ÷ the number at the start

Why the two forms agree: the average of all is the survivors' and the dead ones' averages, each weighted by its share, R̄all = (1 − d) × R̄S + d × R̄D. Subtracting that from R̄S: R̄S − (1 − d) × R̄S − d × R̄D = d × (R̄S − R̄D), the second form. The bias grows with how many died and how far they fell behind the survivors.

Worked example: the lesson's 500 stocks

Survivors +184.7%, all 500 +86.3%: bias = 184.7 − 86.3 = 98.4 points. The 189 delisted ones averaged −75.5% ($100 became $24.46). Second form: d = 189 ÷ 500 = 0.378, and 0.378 × (184.7 − (−75.5)) = 0.378 × 260.2 = 98.4. The same answer.

Holds whenBreaks when

Some leave for good reasons, such as another company buying them at a higher price: then the bias is smaller. A rule that bets on prices falling would have made its best trades on the dead ones, so for it the bias flips and the survivors make it look worse.

Formula: the honest average

The average on the list as it stood at the start is every stock's return, dead or alive, added up and divided by the number that existed.

R̄all = (NS × R̄S + ND × R̄D) ÷ (NS + ND)
NS, ND
how many survived, how many died
R̄S, R̄D
their average returns, the dead ones' up to their last price
R̄all
the honest average: what the plan made on the list as it stood at the start
Worked example: the lesson's 500 stocks

(311 × 184.7 + 189 × (−75.5)) ÷ 500 = (57,441.7 − 14,269.5) ÷ 500 = 43,172.2 ÷ 500 = +86.3%, the result with all 500 switched on.

Holds whenBreaks when

The data has quietly lost the dead ones' last prices or rows (lesson B4, Data quality). Or the list was picked after seeing results, such as keeping the settings that tested best (lesson B7, The best of many).

One more worked example

Made-up numbers: 50 stocks

50 stocks traded at the start. The 35 survivors returned +40% on average; the 15 delisted ones −60%.

Share that died: d = 15 ÷ 50 = 0.3.

Bias = 0.3 × (40 − (−60)) = 0.3 × 100 = 30 points.

Check with the honest average: (35 × 40 + 15 × (−60)) ÷ 50 = (1,400 − 900) ÷ 50 = 500 ÷ 50 = +10%. Today's list shows +40%, and 40 − 10 = 30.

Common mistakes

Check yourself

  1. 200 funds started ten years ago and 50 were closed. The 150 still open returned +30% on average, the 50 closed ones −50%. By how many percentage points does a table of today's funds flatter the average?
    Answerd = 50 ÷ 200 = 0.25. Bias = d × (R̄S − R̄D) = 0.25 × (30 − (−50)) = 0.25 × 80 = 20 points. Check: all 200 averaged (150 × 30 + 50 × (−50)) ÷ 200 = (4,500 − 2,500) ÷ 200 = +10%, and 30 − 10 = 20.
  2. 25 stocks traded at the start. The 20 survivors returned +10% on average, the 5 delisted ones −60%. What is the honest average of all 25?
    Answer(NS × R̄S + ND × R̄D) ÷ (NS + ND) = (20 × 10 + 5 × (−60)) ÷ 25 = (200 − 300) ÷ 25 = −4%. Today's list would have shown +10%.
  3. A website shows every fund in a category that is "currently available", with a ten-year average of +9% a year. Is the +9% probably too high or too low, and why?
    AnswerProbably too high. "Currently available" means the funds still open. Funds that were closed, often for doing badly, are missing, so it is a survivors' average. The honest number needs every fund that was open ten years ago.
  4. In the lesson's 500 stocks, why does taking the delisted ones out raise the average?
    AnswerAll 189 delisted stocks ended below $25, far below the average of all 500 ($186.34). Taking numbers below an average out of it raises the average, and the list dropped them for doing badly, not at random.
  5. A seller's page shows three strategies, each up more than 40% since it began. An old copy of the page listed twelve. What can you say about the seller's strategies as a whole?
    AnswerLittle, until you see what the other nine did. The three shown are the survivors of twelve, and strategies that lost were likely taken off the page. The honest number is the average of all twelve.

Try it yourself

Run the simulation. Every number in this lesson comes from one Python script (below) that uses the course's shared simulated market, SimMarket, and its 500-stock universe with a fixed seed. Change the delisting line, the number of stocks or the plan and see how the bias moves.

Do it by hand. Write down ten made-up stocks with what $100 became in each. Cross out every one below $25 and average the rest; then average all ten. The difference is the bias. Try crossing out any other bottom group: the average always rises.

The script needs simmarket.py, the course's simulated market, in the same folder.

The code behind every number (Python, SimMarket seed 20261109)
Loading…

Going further: the same bias, measured another way

This lesson bought once and held. Another common plan keeps the same money in every stock still trading, topping up the losers and trimming the winners each day. On the same 500 stocks, that plan grows 6.83% a year on all 500 and 14.23% a year on today's 311: SimMarket's own summary measures exactly this. The bias shows in every plan that holds stocks; only its size changes.

For the preciseUniverse: SimMarket universe(500), seed 20261109 (its stocks' stream default_rng([20261109, 1000])), 2,607 trading days from 2015-01-05 to 2024-12-31. Each stock moves with SimMarket's Market 1 times a beta drawn from 0.5 to 1.5, plus Student-t noise of its own (daily volatility 1.5% to 4%) that averages zero; first prices $10 to $100. Delisted on the first close below a quarter of the first price; sold at that close. Buy and hold, $100 a stock at its first price, no costs. Today's list: 311 stocks, $100 became $284.71 on average (+184.7%; 11.0% a year over 9.988 years), median $136.65; 196 ended above $100, 115 at or below; the weakest survivor $29.68, the best $6,505.21. Delisted: 189 (37.8%), $100 became $20.98 to $24.99, average $24.46 (−75.5%); by year 2015–2024: 4, 20, 24, 17, 16, 17, 49, 11, 17, 14. All 500: average $186.34 (+86.3%; 6.4% a year), median $66.45. Bias 98.4 points = 0.378 × 260.2. Kept equal each day (daily average move of the stocks trading that day, compounded): 6.83% a year on all 500, 14.23% on the 311.

Honest notes

Where this comes from