Backtesting Essentials · Lesson 15
A research workflow
Write the hypothesis and the test before the first run, log every version, and let one held-out run give the verdict. The order keeps your own choices from deciding the result.
Know the basics? Read the lesson notesFrom a question to a hypothesis
Research starts with a question. Here it is about a simulated market that publishes a made-up number each evening, the reading. Does the reading say which way the next day goes?
In half the simulated markets of this lesson it does: the next day's move gets an extra 0.10% for each unit of the reading (the reading usually lands between −2 and +2; a reading of +1 adds 0.10%, and a typical day moves about 1.2%). In the other half the reading moves nothing. You will work on one market without being told which kind it is.
A question can't turn out wrong. A hypothesis can: it names the rule, the data, a fair yardstick (something only a real effect would beat; the next card makes it exact) and the result that would count against it. Which of these is a hypothesis?
Written in full, before any run: holding the market for a day only after a reading above 0 beats a fair yardstick on years set aside; an error bar on those years that reaches $0 or below counts against it. An error bar is a range of plausible values for a result, worked out when the test runs. Without that last clause, any result can be read as support.
The exact rule, its costs and its yardstick
The rule: at each close, when the reading comes out, hold $100 until the next close if the reading is above 0; otherwise stay out. Each trade costs 2 cents, about the gap between the buying and selling price at the close in this market. Every result in this lesson is after costs. Written down before any run, the rule and its costs can't drift toward whatever worked.
The yardstick. These markets rise on average, so a rule that sits out some days trails just holding for that reason alone (the lesson Benchmarks). The hypothesis is about which way the next day goes, so the rule is compared with holding's share: just holding's result times the share of days the rule is in. The rule's result minus holding's share is its excess.
Over four years just holding made $50. A rule was in the market on 40 of every 100 days and made $26 after costs. What is its excess?
The split and the line, before any run
Eight simulated years exist, 2015 to 2022. The plan splits them before the first run: working years, 2015–2018, where versions are tried and chosen, and held-out years, 2019–2022, kept for one run of one chosen version (the lesson The held-out test). The held-out years stay out of every choice.
The line: a finding only if the excess's error bar sits wholly above $0. The bar reaches 1.96 standard errors either side of the excess, a standard error being how much the excess would wobble from luck alone. A version with nothing in it, whose excess averages $0, clears that line about 2.5 times in 100 (the held-out card shows why).
The plan, written before any run
Rule: in from the close when the reading is above 0. Costs: 2 cents a trade.
Working years: 2015–2018. Held-out years: 2019–2022, one run.
Line: error bar wholly above $0. The pick: the logged version with the most dollars of excess on the working years.
Once the plan is written, which of these may still change without spoiling the test?
Every version goes in the log
On the working years you may try versions of the rule: the level the reading must clear (−0.5, 0, 0.5 or 1) and an extra condition (none, not on Mondays, not on Fridays, only after a down day, only after an up day). Four levels and five conditions make 20 versions. Every version you run goes in the version log, kept or not.
The more versions, the more chances that one clears the line by luck. The lesson The best of many gave that chance for n tries with nothing in them, each clearing the line with chance p: 1 − (1 − p)n. Each version's z is how many standard errors its excess sits above zero, so the line is z above 1.96. Run five versions of your choice and watch the chance grow.
Your version log, on the working years 2015–2018 (simulated, after costs)
Each dot: one version's z, in the order run. Dashed: the line, 1.96.
With all 20 logged, the chance that at least one version with nothing in it clears 1.96 by luck is 1 − 0.97520 = 0.397, about 40 in 100, where 0.975 = 1 − 0.025 is the chance one version misses. On this market none did: the highest z was 1.57.
The plan's pick goes by dollars of excess, not by the highest z, because that is the rule the plan fixed in advance: version 17, readings above 1, not on Mondays, with $19 (z = 1.46). What the log tells you: even a pass from the best of 20 would have been fairly likely from luck alone, so the pick still faces its one held-out run.
For the precise
Version 19 (readings above 1, only after a down day) had the highest z but less excess. Version 4 (readings above 0, only after a down day) also rounds to $19; in cents version 17 led, 1,930 cents against 1,852. The working years hold 1,039 trading days.One run on the held-out years, and a verdict
The plan sends the pick to the held-out years once, settings unchanged. Will it clear the line?
The pick on 2019–2022: its excess, added up day by day (simulated, after costs)
Orange: the running excess. Bar at the end: its error bar. Dashed: $0.
You guessed a finding. There was none.
You guessed no finding, and there was none.
Over the 1,044 held-out days the pick's excess was −$8. One day's excess swung by σ = $0.36, the typical size of one day's swing (its standard deviation). That is the swing of the excess, not of the price: about a third of a typical day's move of about $1.20, because on most days the excess is only a small share of the move. √1,044 = 32.3, so the standard error is 0.36 × 32.3 = 11.6 and the bar 1.96 × 11.6 = 22.7, about $23 either side: from −8 − 23 = −$31 to −8 + 23 = $15. The bar reaches $0, so the plan says: no finding. Why 1.96: a range that wide misses the excess the rule would average over endless days only 5 times in 100, 2.5 on each side, which is where the line's 2.5 in 100 comes from.
The verdict carries its uncertainty: "no finding; excess −$8 ± $23 after costs on 2019–2022." It does not say the reading is useless. Here it was: this market had its effect switched off. That is one market, and the next card runs the plan on 1,000.
Two ways of working, on 1,000 markets
Picture two researchers on each of 1,000 simulated markets. Each market is built twice: with nothing to find, and with the small effect from the first card.
The planner follows the plan: logs all 20 versions on the working years and runs the pick once on the held-out years. The tweaker sets nothing aside: runs version 1 on all eight years, and if it misses the line, the next version, stopping at the first that clears it and reporting only that one.
With nothing to find, how many of the 1,000 tweakers end with a finding?
Findings in 1,000 research runs (simulated, after costs)
You guessed about 25. It was 112.
You guessed about 110, and it was 112.
You guessed about 400. It was 112.
With nothing to find, 15 planners ended with a finding (2.5 in 100 would give about 25 in 1,000; 15 is this run's count) and 112 tweakers did, about seven to eight times as many.
With the small effect, the tweaker found it in 968 and the planner in 376. The tweaker had twice the data (all eight years against the planner's four held-out years) and kept trying until something passed, and both help it find a real effect. So it finds more real effects and about seven to eight times as many false ones, and from inside its own log it can't tell which of its findings is which. How many of them are luck depends on how common real effects are: here half the markets had one; if real effects were rare, many of the tweaker's findings would be luck. The planner's cost is missing a small real effect more often.
For the precise
Every count is out of 1,000 of this simulation's markets, the same markets in both worlds (the same draws with the effect switched off or on). The reading is a standard normal draw. With nothing to find, a single version cleared the line on the held-out years in 302 of 20,000 version runs, and a tweaker who ended with a finding had run 10.3 versions on average. With the effect, the tweaker's first version cleared the line in 774 and the planner's held-out run missed it in 624.The version log and the chance of a lucky finding
A version with nothing in it that clears the line anyway is a false positive (the lesson The best of many). Trying many versions and keeping the best is multiple testing. The version log makes the chance of a false positive countable: n is every version run, kept or dropped.
- p
- the chance one version with nothing in it clears the line: 0.025 for z above 1.96, when its excess averages $0
- n
- the number of versions in the log, every one run, kept or dropped
- (1 − p)n
- the chance all n miss: 1 − p multiplied by itself n times
n = 20, p = 0.025. One version misses with chance 0.975; all 20 miss with chance 0.97520 = 0.603; at least one clears with 1 − 0.603 = 0.397, about 40 in 100. So a pass from the best of this log is weak evidence until the held-out run.
- no version has anything in it (the formula counts lucky passes);
- each clears the line with the same chance p;
- the versions are separate tries: one's luck says nothing about another's;
- n counts every version, kept or dropped.
The versions are alike: sharing most of their days, they are fewer than n separate tries, so the formula overstates the chance. Here the tweakers ended with a finding in 112 of 1,000, well under the formula's 397 for 20 separate tries at p = 0.025: their versions are related, and costs made each pass less often (the box below gives the numbers). Raising the line to match the log, for related versions too, is the topic of the lesson Many tests at once.
For the precise
With nothing to find, one version cleared the line on the eight years in 208 of 20,000 runs, p ≈ 0.01 (under 0.025, because costs pull its excess below $0). For the tweakers' 20 versions the formula, with n = 20 and that p, gives 0.9920 = 0.82 and 1 − 0.82 = 0.18, or 180 tweakers in 1,000; the count was 112.New numbers. A log of 10 versions, each with p = 0.025. What is the chance at least one clears the line by luck? Type a percentage.
Spot the broken step
Years that helped choose a version are in-sample, and a result on them flatters it. Below are three made-up write-ups. Each breaks one step of the plan. Name it for each.
A: the held-out years helped choose the final version, so they are in-sample and the last run flatters it. B: the write-up hides 27 of the 30 versions, so the chance of a lucky pass looks far smaller than it was. C: the held-out years chose the hypothesis, so the "finding" is a result on in-sample data.
What this means for your research
The steps, in order, each with what goes wrong without it.
- A question.
Without one, you end up testing whatever looked good.
- A hypothesis, with what would count against it.
Including the yardstick. Without it, any result can be read as support.
- The exact rule.
Without it, the rule drifts toward whatever worked.
- Costs, fixed in advance.
Left out, they can hide that a result is gone after trading. Set afterwards, they can be set to keep it.
- The data split, before any run.
Without it, every year has helped choose, and nothing is left to test on.
- A version log.
Without it, nobody knows n, so nobody knows the chance that the best version is luck.
- One run on the held-out years.
A second run, or a second version, makes them in-sample.
- A verdict with its error bar.
Without it, "−$8" reads as a measurement, not as "anywhere from −$31 to $15".
A friend's backtest
If none of the 14 had anything in them, what is the chance that at least one would clear that line by luck? Type a percentage.
Hint: 1 − (1 − p)n with p = 0.025 and n = 14. On a phone calculator: 0.975, the xy key, 14.
Writing the verdict
How many dollars either side of $45 does the error bar reach?
Hint: error bar = excess ± 1.96 × σ × √n, and √900 = 30.
A good-looking version
What is your next step?
Lesson done
A sound workflow: before any run, the question, a hypothesis with the result that would count against it, the exact rule, its costs and the data split are written down. Every version run on the working years is logged, since the count sets the chance that the best of them is luck. One chosen version runs once on the held-out years, and the verdict is written with its error bar. In this lesson's simulation, after costs, with nothing to find, 15 of 1,000 planners and 112 of 1,000 tweakers ended with a finding. With a small planted effect the tweaker found it more often, 968 against 376, with twice the data and by trying until something passed; it also had about seven to eight times as many false findings, and it can't tell which of its findings is which. How much that matters depends on how common real effects are. The planner's cost is missing a small effect more often.
Key ideas
- Hypothesis
- a statement a test can count against: the rule, the data, the yardstick and the result that would make it wrong, written before any run
- Holding's share
- just holding's result times the share of days a rule is in the market: a yardstick that takes out a rising market's average rise
- Excess
- a rule's result minus holding's share, over the same days
- Version log
- a record of every version run, kept or dropped; its count is the n in the chance of a lucky finding
- Finding
- a reported version that clears the line written in the plan
Formula
chance at least one of n clears = 1 − (1 − p)n
For versions with nothing in them, each clearing the line with chance p, as separate tries, with n counting every version run. Alike versions make it overstate the chance.