Quant from Scratch by QuantWright

Backtesting Essentials · Lesson 17

Capstone: one idea, from question to verdict

One question about a simulated market, taken through every step of a backtest you can trust. You make each decision; the page keeps the score, then counts what each shortcut does on 1,000 markets.

Know the basics? Read the lesson notes

A question, then a hypothesis

This lesson takes one question through every step of this course, in order. You make each decision, and the page keeps the score: a point for each careful choice on the first try.

The market is simulated: made-up daily prices and volumes from 2015 to 2022 (volume is the number of shares traded in a day). You stand at the end of 2022. The question: when a day closes up on unusually heavy trading, does that say which way the next day goes? You will learn whether this market has anything to find only after your verdict.

A hypothesis is a statement a test can count against, and it needs a fair yardstick. These simulated markets are built to rise on average, so a rule that holds one on some days tends to make money for that reason alone. So the rule is compared with holding's share: just holding's result (in the market every day) times the share of days the rule is in. The rule's result minus holding's share is its excess. Results are counted after costs, what each trade pays to buy and sell. Which can be tested?

Any result fits "something": a gain, a loss, a pattern on Tuesdays. Nothing could count against it, so nothing can test it.
It names the days, the yardstick and costs. The plan will add the years and the rule that decides it.
It can be tested, but in a market that rises on average, days held tend to make money whatever the volume. It would measure the market's rise. B takes that out.
Your score so far–

What the rule knows, and when

The exact rule. A heavy day: its volume is the highest of the last 10 trading days, itself included. A heavy up day also closes above the day before. After one, the rule holds $100 of the market for 1 day. Those two numbers are settings; you will vary them later.

Look-ahead bias is a backtest using something not yet known when it trades (the lesson Look-ahead bias). A day's close and its full volume are known only when it ends, at 16:00; its open is its first price, at 09:30. When can the rule buy?

That buys at 09:30 on what is known only at 16:00. The backtest would be in only on days already known to close up: it peeks.
An order sent once the close is known fills at the next price there is, the next open. Filling at the close it just read is a same-close fill, which the lesson Inside a backtest engine calls optimistic.
It is the first price after everything the rule read.

So a day's hold runs from one open to the next: it makes $100 × (next open ÷ open − 1).

Your score so far–

Check the data before any run

Before any run, look at the data itself (the lesson Data quality). This is the raw file: prices and volumes exactly as recorded, nothing corrected.

The raw file, 2015–2022 (simulated)

Line: the daily close (left scale, dollars). Bars: volume (lower scale, millions of shares a day).

On 2020-10-05 the close goes from $210.79 the day before to $102.47, and stays near that level. Volume jumps to more than double from then on. What happened? (A 2-for-1 split cuts each share into two; a bad print is one wrong price.)

A crash rarely halves a price exactly and stays there, and volume would not stay higher for good. Those are the marks of a split.
Each share became two worth half as much, so the price halved and twice as many shares change hands. Nothing was lost.
A bad print is a price recorded wrong; it jumps and comes straight back. This one stays.

The clean file is split-adjusted: prices before the split halved and volumes doubled, so the series runs on as one. The split explains a doubling of volume; the rest of the jump is a wild spell, a stretch of large moves, that began around then. In the raw file a day's hold over the split shows −$50.73 on $100, a fall that never happened, and it flows into just holding's result and so into holding's share. The plan uses the clean file.

One market that traded all eight years is tested, so testing only what survived to today (survivorship bias) can't arise here.

Your score so far–

The plan, before any run

Some things are fixed before any result exists, so no result can move them. The years are split in two: working years, where versions of the rule are tried, and held-out years, set aside for one run of one chosen version (the lesson The held-out test).

The line. A result counts as a finding only if its error bar, the range luck could have moved it, sits wholly above $0. That bar is the excess ± 1.96 standard errors, a standard error being how far luck typically moves the excess. An excess with nothing behind it, averaging $0, clears that about 2.5 times in 100 (another 2.5 fall below the bar's low end).

Two choices are yours.

Costs
Years

A spread is the gap between the price to buy and the price to sell; a trade (days in a row in the market) pays it once, 2 cents on $100 at the open here. Left out, costs can hide that a small result is gone after trading. Costs or years chosen after a look can be chosen to fit the results, and if every year helps choose, none is left to test on.

The plan Hypothesis: days after a heavy up day beat holding's share after costs on the held-out years; an error bar there that reaches $0 counts against it.
Rule: buy at the next open after a heavy up day. Data: the clean file. Costs: 2 cents a trade.
Working years: 2015–2018. Held-out years: 2019–2022, one run. Line: z above 1.96.
The pick: the logged version with the most dollars of excess on the working years.
Your score so far–

Nine versions, every one logged

A version is one full set of the rule's settings. Two are open: how many days the volume must top (5, 10 or 20, about a week, two weeks or a month) and how many days to hold (1, 2 or 5). That makes 9. Every version run on the working years goes in the version log, kept or not.

Each version with nothing in it is one more chance to clear the line by luck. With n such versions, each clearing it with chance p, the chance that at least one does is 1 − (1 − p)n (the lesson The best of many). Run all nine.

Your version log on 2015–2018 (simulated, after costs)

Each box: one version's excess and z. Rows: the days the volume must top. Columns: the days held.

hold 1 dayhold 2 dayshold 5 days top of 5 top of 10 top of 20
Versions logged0
Chance at least one clears 1.96 by luck–

With all 9 logged and p = 0.025 (the line's 2.5 in 100), that chance is 1 − 0.9759 = 0.204, about 20 in 100 if the nine were separate tries; fewer, since neighbouring versions share most of their days and so their luck. None cleared 1.96: the highest z was 1.61. The plan's pick, the most dollars of excess, is volume at the top of 5 days, held 5 days: $24 after costs (z = 1.61).

Before the held-out run

Look at the pick's place in the grid. It sits in a corner, and its neighbours, the versions one step away, made $10, −$5 and −$4. A lone high result is a peak; a wide patch of similar ones, a plateau (the lesson Robustness). Neighbours share most of their days, and so their luck: either shape can be luck, and neither replaces the held-out run.

What goes to the held-out years?

The plan chose the pick before any result. The grid's shape goes in the write-up; it doesn't change the test.
Then the held-out years help choose. Years that helped choose are no longer a fair test, and a result on them flatters the choice.
Versions left out of the log hide n, so the chance that the best is luck looks smaller than it is.
Your score so far–

One run on the held-out years

The pick runs once on 2019–2022, settings unchanged. Will it clear the line?

The pick on 2019–2022: its excess, added up day by day (simulated, after costs)

Line: the running excess. Bar at the end: its error bar. Dashed: $0.

Your guess:

Over the 1,043 held-out days the excess was −$23. One day's excess has a standard deviation (its typical distance from its average) of σ = $0.65, more than the $0.46 on the working years, because the held-out years were wilder: a typical day moved $1.32 against $0.96. Luck adds up more slowly than days, with √n: √1,043 = 32.3, so the standard error is 0.65 × 32.3 = 21.0 and the bar reaches 1.96 × 21.0 = 41.2, about $41, either side: from −23 − 41 = −$64 to −23 + 41 = $18. The bar reaches $0: no finding.

A verdict you'd defend

Which verdict would you sign?

Here it happens to be true, as the next lines show, but the test can't tell you so: a small real excess could hide inside a bar $41 wide each way. B says what was measured, and how wide luck is.
It gives what the test measured, with its error bar, its years and its costs, and claims no more.
1.61 is the best of 9 on the years that chose it, which flatters it. The held-out run is the test, and it came out below $0.

Now the answer. This market was built with nothing to find: it comes from the course's simulated-market generator with the one effect it can contain, a link between volume and direction, switched off. Its prices move in calm and wild spells (stretches of small and of large moves), and a day's volume follows how wild the day is, never which way it goes. So no finding was right here. This shows how a careful test behaves when nothing is there; it says nothing about whether real markets have such an effect.

It could have gone wrong either way. With nothing to find, the line is cleared by luck about 2.5 times in 100. And a small real excess could have been missed: a true excess as large as the bar's reach, $41 over these four years, would clear the line only about half the time, because the measured excess lands above its true value about half the time. Had the pick passed, the next evidence would be a forward test: the rule, unchanged, on days that did not exist when it was fixed, with the result that would stop it written down first (the lesson From backtest to forward test).

Your score so far–
Theory

The arithmetic of a verdict

The verdict rests on three lines. The first takes out the market's average rise, for a hypothesis about direction. The other two say how far luck could have moved the result: the error bar is a 95% confidence interval (a range built so that luck alone puts the true value outside it only about 5 times in 100), and z is a z-score: the same excess counted in standard errors, read against 1.96.

excess = R − f × H
error bar = excess ± 1.96 × σ × √n
z = excess ÷ (σ × √n); a finding when z > 1.96
R
the rule's result over the test days, after costs
H
just holding's result over the same days
f
the share of those days the rule is in the market
σ
the standard deviation of one day's excess (its typical distance from its average)
n
the number of test days; σ × √n is the standard error
1.96
the multiplier for a 95% interval
Worked example: the pick on 2019–2022

R = −$23.74, H = −$1.90 (over these four years this market happened to go nowhere), f = 0.372. Holding's share: 0.372 × −1.90 = −$0.71. Excess: −23.74 + 0.71 = −$23.03. Standard error: 0.65 × 32.3 = 21.0, so the bar is ± 41.2 and z = −23 ÷ 21.0 = −1.1: no finding.

Holds when
  • one version, chosen before the run, runs once as planned;
  • one day's excess doesn't change the next day's;
  • enough days that luck in the total is close to the usual bell shape.
Breaks when

The run is the best of several on the same days: the bar then sits around a flattered number (the lesson The best of many). Calm and wild spells, as here, make 95 in 100 approximate.

New numbers. Over 900 test days a rule made R = $40 after costs, in on a quarter of the days, while just holding made $80. One day's excess has σ = $0.50. How many dollars either side does its error bar reach?

√900 = 30, so the standard error is 0.50 × 30 = $15 and the bar 1.96 × 15 = $29.40. The excess is 40 − 0.25 × 80 = $20, so the bar runs from −$9.40 to $49.40. It reaches $0: no finding.
Standard error: σ × √n = 0.50 × 30 = $15. Bar: 1.96 × 15 = $29.40 either side. The excess is 40 − 0.25 × 80 = $20, so the bar runs from −$9.40 to $49.40: it reaches $0, no finding.

Your score, and what each shortcut does

Your scoreAnswer the steps above to see it.
  • A hypothesis with a fair yardstick
  • A rule that reads only what was known
  • The data checked
  • Costs fixed in advance
  • Untouched years kept for one run
  • The pick sent once, unchanged
  • A verdict with its error bar

One market is one draw. The same study ran on 1,000 more markets from the same generator, each with nothing to find, so every finding is a false one. With every choice made carefully, in how many does the plan still end with a finding?

Findings in 1,000 markets with nothing to find (simulated)

Your guess:

The careful plan ended with a finding in 30 of 1,000, near the 25 that 2.5 in 100 gives: the line lets luck through that often by design. The figure shows each shortcut alone, and your path: it starts with the shortcut behind each choice you missed; tap the shortcuts to switch them. No costs and the raw file moved the count only a little (34 and 38 against 30, within chance here): a 2-cent cost is small, and the raw file's fake fall pushes results both ways (the box below). With no untouched years the best of 9 is judged on the years that picked it: 93. Buying at the open uses a price the rule couldn't know yet: 976. All counts are this simulation's, after costs where costs are charged.

For the preciseThe raw file. The fake fall lands in the held-out years of every market. In 833 of 1,000 the pick was out of the market that day, so the fall lowered only holding's share and raised the excess (by $5.45 in the middle market); but that one huge day also widened the error bar (by 15% in the middle market), so z moved little. In the other 167 the pick held through the fall and lost. On the working years both files give the same triggers, so the pick was the same in 972 of 1,000 (the rest differ by cent rounding of the doubled prices).
Practice 1 of 3

A friend's backtest

The situationA friend writes: "Buy at the open on any day that closes up on the month's heaviest volume, sell at the next open. On 2015–2022: $310 after costs, z = 6.2."

What is the first problem to raise?

A real problem, but a smaller one. The rule buys at the open on a close and a volume that are only known at 16:00: it peeks. In this lesson's simulation, buying at the heavy day's open gave a finding in 976 of 1,000 markets with nothing to find.
A rule that peeks can show any z. It buys at the open on a close and a volume known only at 16:00, so it is in only on days already known to close up.
It peeks: the close and the full volume are known only when the day ends. In this lesson's simulation that one shortcut gave a finding in 976 of 1,000 markets with nothing to find.
Practice 2 of 3

Is it a finding?

The situationYour pick ran once, as planned, on 1,600 held-out days. It made $30 after costs, in on a fifth of the days, while just holding made $50. One day's excess had a standard deviation of $0.40.

What is its z?

Hint: excess = R − f × H; standard error = σ × √n, and √1,600 = 40; z = excess ÷ standard error.

Excess: 30 − 0.2 × 50 = $20. Standard error: 0.40 × 40 = $16. z = 20 ÷ 16 = 1.25, below 1.96: no finding. The verdict: "excess $20 ± $31.36 after costs".
Holding's share: 0.2 × 50 = $10, so the excess is 30 − 10 = $20. Standard error: 0.40 × √1,600 = 0.40 × 40 = $16. z = 20 ÷ 16 = 1.25, below 1.96: no finding.
Practice 3 of 3

After a pass

The situationIn another study your pick cleared the line on its held-out years, run once as planned, with 9 versions logged.

What is the next step that adds evidence?

The held-out years have given their one result. A second run there can only repeat it, or, if anything changes, let those years help choose.
New days that did not exist when the rule was fixed are the next evidence, with the result that would stop it decided before the first day.
More versions add chances for luck, and a new pick would need years nothing has touched, which are used up.
✓

Lesson done

One question went through every step: a hypothesis with a fair yardstick, a rule that reads only what was known, checked data, costs and a split fixed before any run, every version logged, one held-out run, and a verdict with its error bar. Here the verdict was no finding, excess −$23 ± $41 after costs, on a simulated market built with nothing to find; that says nothing about whether volume ideas work in real markets, and it does not show that nothing is there, since a small real excess could hide inside the bar. In this lesson's simulation, on 1,000 such markets, the careful plan still ended with a finding in 30, by luck; without untouched years it was 93, and buying before the day's close and volume were known, 976.

Key ideas

Verdict
what a test measured, with its error bar, its years and its costs, and no more; "no finding" is not proof that nothing is there
Raw file, clean file
prices and volumes as recorded, and the same adjusted for events such as a split
Shortcut
a step skipped or done after seeing results; in this lesson's simulation some added false findings only a little (no costs, the raw file) and some a lot (no untouched years, buying before the close was known)

Formulas

excess = R − f × H  ·  error bar = excess ± 1.96 × σ × √n  ·  z = excess ÷ (σ × √n)

For one version chosen before the run and run once as planned, on days whose excesses don't change each other; around the best of several runs on the same days the bar sits on a flattered number.

Lesson notes

A backtest you can trust takes one question through a fixed order: a hypothesis with a fair yardstick, a rule that reads only what was known when it trades, checked data, costs and a split of the years fixed before any run, every version logged, one held-out run of one chosen version, and a verdict with its error bar. Every number below is from the lesson's simulated markets, all built with nothing to find, after costs unless a line says otherwise; they show this simulation, not any real market, and "no finding" is not proof that nothing is there.

The question, the hypothesis and the yardstick

The lesson's market is simulated: daily prices and volumes (shares traded in a day) from 2015 to 2022, with the researcher standing at the end of 2022. The question: when a day closes up on unusually heavy trading, does that say which way the next day goes? A hypothesis is a statement a test can count against. These simulated markets are built to rise on average, so "days after a heavy up day make money" would measure the market's rise. The fair yardstick is holding's share: just holding's result (in the market every day) times the share of days the rule is in. The rule's result minus holding's share is its excess. The hypothesis: days after a heavy up day beat holding's share after costs on the held-out years, and an error bar there that reaches $0 counts against it.

What the rule knows, and the data

A heavy day's volume is the highest of the last 10 trading days, itself included; a heavy up day also closes above the day before. After one, the rule holds $100 for one day. A day's close and full volume are known only at 16:00, so buying at that day's open peeks (look-ahead bias), and filling at the close the rule just read is an optimistic same-close fill. The rule buys at the next open, so a day's hold runs from one open to the next and makes $100 × (next open ÷ open − 1).

Before any run the data were checked. The raw file, prices and volumes as recorded, showed the close going from $210.79 to $102.47 on 2020-10-05, staying there, with volume jumping to more than double from then on: a 2-for-1 split, each share becoming two worth half as much (the split doubles the volume; the rest of the jump is a wild spell, a stretch of large moves, that began around then). In the raw file a day's hold over the split shows −$50.73 on $100, a fall that never happened, which also flows into holding's share. The clean file is split-adjusted, and the plan uses it. One market that traded all eight years was tested, so survivorship bias (testing only what survived to today) can't arise; it would if markets were picked from today's list.

The plan: costs, years and the line

Fixed before any run: costs of one spread a trade (the gap between the price to buy and the price to sell), 2 cents on $100 at the open, where a trade is days in a row in the market; working years 2015–2018 for trying versions and held-out years 2019–2022 for one run of one chosen version; and the line. A result is a finding only if its error bar, the excess ± 1.96 standard errors (a standard error being how far luck typically moves the excess), sits wholly above $0: the excess's z, its distance above $0 in standard errors, must be above 1.96. An excess with nothing behind it, averaging $0, clears that about 2.5 times in 100. Costs or a split chosen after the results can be chosen to fit them.

The version log and the pick

A version is one full set of settings: the days the volume must top (5, 10 or 20) and the days held (1, 2 or 5), 9 in all, each logged. With 9 versions that have nothing in them, each clearing the line with chance 0.025, the chance at least one clears it by luck is 1 − 0.9759 = 0.204, about 20 in 100 if they were separate tries; fewer, since neighbouring versions share most of their days and so their luck. On the working years none cleared 1.96. The pick, the most dollars of excess, was volume at the top of 5 days held 5 days: $24 after costs, z = 1.61, in a corner of the grid with neighbours (versions one step away) at $10, −$5 and −$4. A lone high result is a peak, a wide patch of similar ones a plateau; neighbours share most of their days, so either can be luck. The pick went to the held-out years once, unchanged: running the best few there would let those years help choose.

The held-out run and the verdict

Over the 1,043 held-out days the pick's excess was −$23 after costs. One day's excess had a standard deviation (its typical distance from its average) of $0.65 (against $0.46 on the calmer working years, where a typical day moved $0.96 against $1.32), which gives an error bar of about $41 either side (worked below): from −$64 to $18. It reaches $0, so the verdict is "no finding: excess −$23 ± $41 after costs, 2019–2022". "Heavy up days say nothing" would claim more than one test can show; "a near miss, z = 1.61 on the working years" leans on the best of 9 on the years that chose it. The market was built with nothing to find (the course's simulated-market generator with the one effect it can contain, a link between volume and direction, switched off; there volume follows how wild a day is, never its direction), so no finding was right here. That shows how a careful test behaves when nothing is there and says nothing about whether real markets have such an effect. A true excess as large as the bar's reach, $41 over four years, would clear the line only about half the time. After a pass, the next evidence would be a forward test: the rule, unchanged, on days that did not exist when it was fixed, with the result that would stop it written down first.

Every shortcut on 1,000 markets

The same study ran on 1,000 more simulated markets from the same generator, each with nothing to find, so every finding is a false one. The careful plan ended with a finding in 30, near the 25 that 2.5 in 100 gives. On the same markets, leaving out costs gave 34 and the raw file 38, only a little more and within chance here: a 2-cent cost is small, and the raw file's fake fall pushes results both ways. The fake fall lands in the held-out years of every market. In 833 of 1,000 the pick was out of the market that day, so the fall lowered only holding's share and raised the excess (by $5.45 in the middle market); but that one huge day also widened the error bar (by 15% in the middle market), so z moved little. In the other 167 the pick held through the fall and lost. With no untouched years (the best of 9 on all eight years, judged on the years that picked it) it was 93, and buying at the heavy day's open, a price the rule couldn't know yet, 976. These counts are this simulation's.

Formula: the arithmetic of a verdict

The first line takes out the market's average rise, for a hypothesis about direction; the other two say how far luck could have moved the result. The error bar is a 95% confidence interval and z a z-score.

excess = R − f × H
error bar = excess ± 1.96 × σ × √n
z = excess ÷ (σ × √n); a finding when z > 1.96
R
the rule's result over the test days, after costs
H
just holding's result over the same days
f
the share of those days the rule is in the market
σ
the standard deviation of one day's excess (its typical distance from its average)
n
the number of test days; σ × √n is the standard error
1.96
the multiplier for a 95% interval
Worked example: the pick on 2019–2022

R = −$23.74, H = −$1.90 (over these four years this market happened to go nowhere), f = 0.372. Holding's share: 0.372 × −1.90 = −$0.71. Excess: −23.74 + 0.71 = −$23.03. Standard error: 0.65 × 32.3 = 21.0, so the bar is ± 41.2 and z = −23 ÷ 21.0 = −1.1: no finding.

Holds whenBreaks when

The run is the best of several on the same days: the bar then sits around a flattered number. Calm and wild spells, as in these markets, make 95 in 100 approximate.

One more worked example

Made-up round numbers

A pick runs once on 1,600 held-out days. It makes $18 after costs, in on a tenth of the days, while just holding makes $60; one day's excess has σ = $0.30. Holding's share: 0.1 × 60 = $6, so the excess is 18 − 6 = $12. √1,600 = 40, so the standard error is 0.30 × 40 = $12 and z = 12 ÷ 12 = 1.0. The bar reaches 1.96 × 12 = $23.5 either side, from −$11.5 to $35.5: it reaches $0, so the verdict is "no finding: excess $12 ± $23.50 after costs". Against $0 instead of holding's share, the same rule would have shown $18.

Common mistakes

Check yourself

  1. Why is the rule compared with holding's share and not with $0?
    Answer

    The markets are built to rise on average, so a rule that holds one on some days tends to make money for that reason alone. The hypothesis is about which way the next day goes, so the yardstick takes out the rise: just holding's result times the share of days the rule is in.

  2. A rule reads a day's close and its full volume. When is the first price it can honestly buy at?
    Answer

    The next day's open. Both are known only when the day ends at 16:00; buying at that day's open peeks, and filling at the close it just read is an optimistic same-close fill.

  3. A raw price file shows the close halving in one day and staying there, with volume doubled from then on. What is it, and what do you do?
    Answer

    A 2-for-1 split: each share became two worth half as much. Use the split-adjusted (clean) file, in which prices before the split are halved and volumes doubled; the raw file shows a fall that never happened.

  4. A pick runs once on 900 held-out days. It makes $25 after costs, in on half the days, while just holding makes $30; one day's excess has σ = $0.40. Is it a finding?
    Answer

    Holding's share: 0.5 × 30 = $15, so the excess is 25 − 15 = $10. Standard error: 0.40 × √900 = 0.40 × 30 = $12, so z = 10 ÷ 12 = 0.83, below 1.96: no finding, "excess $10 ± $23.52 after costs", for one version chosen before the run and run once.

  5. In the lesson's simulation, on 1,000 markets with nothing to find, how many times did the careful plan end with a finding, and how many without untouched years?
    Answer

    30 and 93 of 1,000. The line lets luck through about 2.5 times in 100 by design; keeping the best of 9 on years that also judge it lets through more.

Try it yourself

Run the script. Every number in this lesson comes from one Python script (below). It builds the lesson's simulated market in its clean and raw files, measures the cost, runs the nine versions on the working years, makes the pick, runs it once on the held-out years, and puts all 16 combinations of the four shortcuts through 1,000 more markets. Change a setting, the line or the years, and see what moves.

By hand, in a spreadsheet. Write the plan first, dated: the question, the hypothesis and what counts against it, the rule and when it trades, the data file, the costs, the working and held-out years, the line. Keep a second sheet as the log, one row per version run, never deleted. For the verdict, a column of one-day excesses gives the rest: =SUM(col) is the excess, =STDEV.P(col)*SQRT(COUNT(col)) its standard error, and the bar is 1.96 times that either side.

The script needs simmarket.py, the course's simulated market, in the same folder.

The code behind every number (Python, on the course's simulated-market generator; market seeds 2026171001 to 2026172001)
Loading…

Honest notes

Where this comes from