Quant from Scratch by QuantWright

Testing Popular Ideas · Lesson 1

How we test an idea

Five checks to run before you believe that a trading idea worked on past prices. Every later lesson in this course uses them.

Know the basics? Read the lesson notes
Check 1 of 5

Write the exact rule

Here is a trading idea you might read online: “buy when momentum turns”. Momentum is how much the price has changed over the last few days, in percent. The 10-day change compares today's price with the price 10 days ago.

To test an idea, a computer runs it over past prices and lists what it would have bought and sold; each buy with its later sale is one trade. That run is a backtest. A computer needs exact instructions, and this sentence fits more than one set. Here are three:

Below is one year of a simulated market: prices made up by a computer, not a real market. Switch between the rules.

One simulated year (2016), and the trades each rule made

Line: the price at the end of each day. Dot: a buy. Shaded: the days from that buy to its sale.

Show
Trades that year24

Rule A traded 24 times that year, Rule B 40 times and Rule C 6 times. One sentence gave three different sets of trades.

An exact rule says when to buy, when to sell, and how much money goes into each trade. The test: two people who code it get the same trades.

For the preciseThe details these rules fix: a rule decides at the end of a day and trades at the first price of the next day, because nobody can trade at a price they have only just seen. Every trade puts in the same 100. A rule holds one position at a time.
Check 2 of 5

Take off the costs

Take Rule B over eight simulated years, 2015 to 2022. Every trade puts in the same 100, one trade at a time, and we add up what the trades made and lost. They add up to +95: a result of +95%, or 95 points (a point is 1 on the 100).

That is before costs. Every trade pays a cost: the broker's fee, plus a slightly worse price than the chart shows, both when buying and when selling. Here one trade's cost is about 0.135% of the 100, so 0.135 points.

Rule B's account on the simulated market, 2015–2022

It starts at 100 and adds each trade's result. Where the line is flat, the rule held nothing.

Trades298
Cost of one trade0.135 points
Costs in all?
Result+95%

Rule B made 298 trades. How many of its 95 points do you think the costs took?

About 1 point is what 8 trades pay (8 × 0.135 = 1.08). Rule B made 298.
About 13 points is what 100 trades pay (100 × 0.135 = 13.5). Rule B made 298.
A small cost, paid 298 times.
298 × 0.135 ≈ 40, so the costs took about 40 points and left 95 − 40 = +55%. A result before costs is called gross; after costs, net. The chart now shows both.
Check 3 of 5

Compare with holding

+55% after costs sounds good. Compared with what, though? The simplest alternative needs no rule: buy on the first day of 2015, sell on the last day of 2022, and pay the cost once. Call it just holding. It is the benchmark: a simple alternative that a rule is measured against.

So the number this lesson reads is the rule's result after costs, against just holding over the same days.

Rule B against just holding, simulated, 2015–2022

Both after costs, both starting at 100. Holding's line appears once you guess.

What do you think just holding made over those eight years?

It made much more than that.
The same as the rule? It made more.
Yes, about that.
Just holding made +85%, so Rule B's +55% is 30 points behind. That gap is a rule's lead over holding; here it is −30. This market rose, and Rule B held nothing part of the time, so it missed part of the rise. The question to ask of a backtest is “did it beat just holding, after costs?”, not “did it make money?”
Check 4 of 5

Count the versions you try

Rule B lost to holding. The tempting next move is to change the settings and try again. The number of days it looks back can be 5, 10, 20, 30, 40, 50, 60, 90, 120 or 200: 10 choices. “Turns” can mean turning positive, or rising after a fall: 2 choices. The sale can come when the change turns back, after 5 days or after 20 days: 3 choices.

That makes 10 × 2 × 3 = 60 versions: the same idea with different settings. Rules A, B and C are the first three on the slider. Slide to try all 60.

60 versions of one idea on the simulated market, 2015–2022

Each dot: one version's result after costs. Dashed line: just holding, +85%. A dot above the line beat holding.

Best so far–
Its result after costs–
Beat holding–

2 of the 60 beat holding, with +88% and +87%. The best one buys when the 120-day change rises after a fall and sells 5 days later. It made +88% after costs: 3 points ahead of holding's +85%.

Shown on its own, that version looks like a find.

For the preciseExact results. The best version: 267 trades, +123.8% gross, 35.9 points of costs, +87.9% net, 3.1 points ahead of holding's +84.8%. Rule B: +94.8% gross, 40.1 points of costs, +54.7% net, 30.1 points behind. Before costs, 6 of the 60 beat holding's +85.0%; after costs, 2. The cost of one trade differs a little between versions, from 0.133% to 0.143%.
Check 4 of 5

What put the best one ahead?

Here is a fact about this simulated market. We built it so that past prices say nothing about which way the next move goes: it rises slowly on average, and the rest of each day's move is chance. So no rule that reads past prices can tell the next move here. We say such a rule has no skill.

What put the best version 3 points ahead of holding?

There are no right settings here. Past prices say nothing about which way the next move goes, so no setting can read it. On other years, another version would come out on top.
With nothing to find, a version's result is what the market paid while it was invested, minus its costs, plus or minus luck. The top of 60 is the luckiest of 60.
No, it paid more than most: it traded 267 times, and 40 of the other 59 versions traded less often and paid less. What set it apart was luck.
That is why the count matters. Try enough versions and one is likely to beat holding by luck alone. A fair report says “the best of 60”, not just “+88%”. So count every version you try, the dropped ones too. In a real market you can't know in advance that there is nothing to find, which is what the count and the next check are for.
Check 5 of 5

Keep some years held out, and use them once

How can you tell luck from something real? Luck rarely repeats. So before running anything, we set two more years aside: 2023 and 2024. No version has seen them. Years kept aside like this are held out.

Now the best of the 60 runs on them, once. Same comparison: its result after costs against just holding. Guess first, then run it.

The best of 60 on the held-out years, simulated, 2023–2024

Both after costs, both starting at 100.

Your guess: the rule ends
2015–2022: rule and holding+88% and +85%
2023–2024: the rule?
2023–2024: holding?

On the held-out years the rule lost 26% over 76 trades, while just holding ended almost where it started, at −0.2%. The lead did not carry over: the search picked this version for its luck on 2015–2022, and luck doesn't repeat. This time luck ran against it (−16% before costs), and the cost of 76 trades took about 10 points more.

One held-out run is a check, not proof, either way: luck can pass a rule with nothing in it and fail one with something. But a version picked for its luck is the kind you expect to fail, and this one did.

A held-out test counts once. If you change the rule and run those years again, they have helped choose the rule, and they are no longer held out (lesson B10, The held-out test).

For the preciseExact: −26.3% after costs over 76 trades, −16.2% before costs; holding −0.2%. On 2023–2024, 17 of the 60 versions beat holding, and the best of 2015–2022 ranked 55th of 60.
Theory

Net result, and the lead over holding

Two subtractions turn a backtest's headline number into the one that counts. First the costs: the number of trades times the cost of one. Then just holding, the benchmark.

net = gross − n × c
lead = net − hold
gross
the trades' results added up, before costs, in points
n
the number of trades
c
the cost of one trade, in points
net
the result after costs, in points
hold
what just holding made over the same days, after its one cost
lead
points ahead of holding; a minus sign means behind
Worked example: your Rule B

net = 95 − 298 × 0.135 ≈ 95 − 40 = 55.

lead = 55 − 85 = −30: 30 points behind holding.

Holds when
  • every trade puts in the same amount, and results are added up;
  • every trade pays about the same cost;
  • holding covers the same days, after costs.
Breaks when

The amount traded grows with the account: results then multiply instead of adding (lesson A14, Compounding and the asymmetry of losses).

A rule made 70 points gross over 150 trades. One trade costs 0.2 points. Just holding made 25 points over the same days. What is the rule's lead over holding? Type a number.

net = 70 − 150 × 0.2 = 70 − 30 = 40. lead = 40 − 25 = 15 points ahead.
Costs first: 150 × 0.2 = 30, so net = 70 − 30 = 40. Then holding: lead = 40 − 25 = 15 points ahead.
Theory

More versions, a stricter pass mark

Trying many versions and keeping the best is called multiple testing. Each extra version gives luck one more try.

First decide how often you accept being fooled by luck: say 5 times in 100. Call that α (alpha). For one rule tested alone, the pass mark is then a lead over holding so large that a rule with no skill reaches it in only 5 of every 100 tests. Put 60 no-skill versions against that same mark and expect 60 × 5 ÷ 100 = 3 lucky passes. The fix: give each version a stricter mark, a larger lead that a no-skill version reaches only α ÷ n of the time. Then a lucky pass anywhere among the n versions happens at most n × α ÷ n = α of the time: 5 in 100 for the whole search. This is the Bonferroni correction.

chance per version = α ÷ n
chance per version
how often a no-skill version may reach the mark set for it
α
how often you accept a lucky pass anywhere in the whole search, often 5 in 100
n
the number of versions tried, dropped ones included
Worked example: your search

α = 5 in 100 and n = 60: 5 ÷ 60 ≈ 0.083 in 100 for each version, which is 1 time in 1,200.

You have already seen how often luck gets as far as the best version did: on this market, where no version has skill, the best of 60 led holding by 3 points. A mark that luck may reach only 1 time in 1,200 has to sit much further out. How far, in points, is lesson B6 (Hypothesis tests).

Holds when
  • n counts every version you looked at, kept or dropped;
  • the versions are separate tries.
Breaks when

Versions are close relatives: a 120-day and a 200-day look-back trade on many of the same days, so their results move together and α ÷ n is stricter than needed (lesson C5, Many tests at once).

For the preciseHow to find how often luck reaches a given result is lesson B6 (Hypothesis tests); searches are lesson B7 (The best of many). The chance that at least one of n separate no-skill versions passes a mark that each passes with chance p is 1 − (1 − p)n. For p = 0.05 and n = 60: 0.9560 = 0.046, and 1 − 0.046 = 0.954, about 95 in 100.

Someone tried 25 versions and accepts α = 5 in 100 for the whole search. How often may luck reach the pass mark for each version?

α ÷ n = 5 ÷ 25 = 0.2 in 100: 1 time in 500.
That is the mark for one version tested alone. With 25 versions it lets through 25 × 5 ÷ 100 = 1.25 lucky passes. Divide instead: 5 ÷ 25 = 0.2 in 100.
Multiplying makes the mark easier, and nothing happens 125 times in 100. Divide instead: 5 ÷ 25 = 0.2 in 100.

The five checks

Every later lesson in this course puts a popular idea through these five checks, in this order.

  1. Write the exact rule.

    Two people who code it must get the same trades.

Practice 1 of 3

Same idea, different results

The situationA post says: “Buy when momentum turns up, sell when it fades.” You and a friend each code it and backtest it on the same prices. Yours makes 40 trades, your friend's makes 90.

What is the most likely reason?

Possible, but not the likely cause. “Momentum” needs a number of days, and “turns up” and “fades” can each be read several ways, so two careful coders end up with two rules.
A backtest on the same prices gives the same trades every time. The difference is in the rules: “momentum”, “turns up” and “fades” can each be read several ways.
“Momentum” needs a number of days, and “turns up” and “fades” can each be read several ways. You coded two different rules, so you got two different backtests.
Write both rules out exactly, and count them as two versions.
Practice 2 of 3

A friend's backtest

The situationA friend's rule made 48 points gross over 120 trades, 2016 to 2023. One trade costs 0.15 points. Just holding made 20 points over the same days, after costs.

What is the rule's lead over holding? Type a number.

Hint: net = gross − trades × cost of one trade. Then lead = net − hold.

net = 48 − 120 × 0.15 = 48 − 18 = 30. lead = 30 − 20 = 10 points ahead.
Costs: 120 × 0.15 = 18, so net = 48 − 18 = 30. Then lead = 30 − 20 = 10 points ahead of holding.
That is checks 2 and 3. Two are still open: how many versions your friend tried, and what held-out years show.
Practice 3 of 3

A second look at the held-out years

The situationYou try 30 versions on 2015–2022 and run the best one on held-out 2023–2024. It ends behind holding. You change one setting, run 2023–2024 again, and now it ends ahead.

Does the second run count as a held-out test?

The first rule was. The second was changed because of what 2023–2024 showed, so those years helped build it.
The setting was changed after seeing those years, so they are now part of the search. They can't also be its fair test.
The size of the change isn't the point. It was made because of what 2023–2024 showed, so those years helped choose the rule.
Count the changed rule as version 31. A fair test of it needs years that nothing has touched.
✓

Lesson done

You can now put a trading idea through five checks: exact rule, costs, just holding, versions counted, held-out years used once.

Key ideas

Backtest
running a rule over past prices to see its trades
Exact rule
one that two people would code into the same trades
Gross, net
a result before costs; after costs
Just holding
buy on the first day, sell on the last: the benchmark a rule is measured against
Version
the same idea with different settings; count every one
Multiple testing
trying many versions and keeping the best; the Bonferroni correction sets a pass mark that luck reaches n times less often
Held-out years
years set aside before you start, run once on the final rule

Formulas

net = gross − n × c  ·  lead = net − hold

chance per version = α ÷ n

Lesson notes

A backtest is a computer running a trading rule over past prices and listing its trades; each buy with its later sale is one trade. A good backtest number is not yet evidence. Five checks make it a result worth taking seriously: an exact rule, costs taken off, a comparison with just holding, the versions tried counted, and some years kept aside (held-out years) and used once.

Check 1: write the exact rule

Take the sentence “buy when momentum turns”. Momentum is how much the price has changed over the last few days, in percent; the 10-day change compares today's price with its price 10 days ago. The sentence fits several rules. Rule A: buy when the 10-day change turns from negative to positive, sell when it is no longer positive. Rule B: buy when the 10-day change has been falling and then starts to rise, sell 5 days later. Rule C: buy when the 20-day change turns from negative to positive, sell 20 days later.

On one year (2016) of the lesson's simulated market, prices made up by a computer, the simulation gave 24, 40 and 6 trades for Rules A, B and C. An exact rule says when to buy, when to sell and how much goes into each trade; two people who code it get the same trades. Here every trade puts in the same 100.

Check 2: take off the costs

Every trade pays a cost: the broker's fee, plus a slightly worse price than the chart shows, on the buy and on the sale. The lesson uses about 0.135% of the 100 for one trade, the buy and the sale together, so 0.135 points (a point is 1 on the 100). Over eight simulated years, 2015 to 2022, Rule B made 298 trades that added up to 95 points, +95%, before costs (gross). These are points on a single 100, added up, not an account that compounds. The costs: 298 × 0.135 ≈ 40 points. After costs (net): 95 − 40 = 55, so +55%.

Check 3: compare with just holding

+55% sounds good, but compared with what? The simplest alternative needs no rule: buy on the first day of 2015, sell on the last day of 2022, pay the cost once. That is just holding, the benchmark: a simple alternative a rule is measured against. On the lesson's simulated market it made +85%, so Rule B's +55% is 30 points behind. That gap is the rule's lead over holding, here −30. The market rose, and Rule B held nothing part of the time. Ask of a backtest “did it beat just holding, after costs?”, not “did it make money?”

Check 4: count the versions you try

Rule B lost to holding, so the tempting next move is to change its settings. A version is one full set of choices: the look-back (10 choices from 5 to 200 days), what “turns” means (turning positive, or rising after a fall: 2 choices) and when to sell (when the change turns back, after 5 days or after 20 days: 3 choices). That makes 10 × 2 × 3 = 60 versions.

In the lesson's simulation, 2 of the 60 beat holding after costs, with +88% and +87%. The best one buys when the 120-day change rises after a fall and sells 5 days later: +88% after costs, 3 points ahead of holding's +85%.

This market was built so that past prices say nothing about which way the next move goes: it rises slowly on average and the rest is chance. A rule that reads past prices has no skill here, so the 3-point lead was luck: the top of 60 tries is the luckiest of 60. Costs did not explain it: it made 267 trades, and 40 of the other 59 versions traded less often and paid less. On a market with nothing to find, try enough versions and one is likely to beat holding by luck alone. So a fair report says “the best of 60”, not just “+88%”, and counts every version tried, dropped ones too. In a real market you can't know there is nothing to find, which is what the count and the next check are for.

Check 5: keep some years held out, and use them once

Luck rarely repeats. So the lesson set aside 2023 and 2024, which no version had seen. The best of the 60 ran on them once. In the lesson's simulation it lost 26% over 76 trades, while just holding ended at −0.2%. The lead did not carry over: the search picked this version for its luck. Luck ran against it (−16% before costs), and the cost of 76 trades took about 10 points more.

One held-out run is a check, not proof, either way: luck can pass a rule with nothing in it and fail one with something. But a version picked for its luck is the kind you expect to fail. A held-out test counts once: if you change the rule and run those years again, they have helped choose the rule and are no longer held out.

Formula: net result, and the lead over holding

net = gross − n × c
lead = net − hold
gross
the trades' results added up, before costs, in points
n
the number of trades
c
the cost of one trade, in points
net
the result after costs, in points
hold
what just holding made over the same days, after its one cost
lead
points ahead of holding; a minus sign means behind
Worked example: the lesson's Rule B

net = 95 − 298 × 0.135 ≈ 95 − 40 = 55.

lead = 55 − 85 = −30: 30 points behind holding.

Holds whenBreaks when

The amount traded grows with the account: results then multiply instead of adding (lesson A14, Compounding and the asymmetry of losses).

Formula: more versions, a stricter pass mark

Trying many versions and keeping the best is multiple testing. Decide how often you accept being fooled by luck, say 5 times in 100: α (alpha). For one rule tested alone, the pass mark is a lead over holding, in points, so large that a no-skill rule reaches it in only 5 of 100 tests. Put 60 no-skill versions against that mark and expect 60 × 5 ÷ 100 = 3 lucky passes. So give each version a stricter mark, one a no-skill version reaches only α ÷ n of the time. A lucky pass anywhere among n versions then happens at most n × α ÷ n = α of the time. This is the Bonferroni correction.

chance per version = α ÷ n
chance per version
how often a no-skill version may reach the mark set for it
α
how often you accept a lucky pass anywhere in the whole search, often 5 in 100
n
the number of versions tried, dropped ones included
Worked example: the lesson's search

α = 5 in 100 and n = 60: 5 ÷ 60 ≈ 0.083 in 100 for each version, which is 1 time in 1,200.

On the lesson's market, where no version has skill, the best of 60 led holding by 3 points. A mark that luck may reach only 1 time in 1,200 has to sit much further out. How far, in points, is worked out in lesson B6 (Hypothesis tests).

Holds whenBreaks when

Versions are close relatives: a 120-day and a 200-day look-back trade on many of the same days, so their results move together and α ÷ n is stricter than needed (lesson C5, Many tests at once).

One more worked example

Made-up round numbers

A rule made 62 points gross over 80 trades, at 0.2 points a trade. Just holding made 30 points over the same days, after its one cost. The rule was the best of 40 versions, and α is 5 in 100.

Costs: 80 × 0.2 = 16 points.

Net: 62 − 16 = 46 points.

Lead: 46 − 30 = 16 points ahead of holding.

Mark for each version: 5 ÷ 40 = 0.125 in 100, which is 1 time in 800.

The +16 is the best of 40 versions, so it still has to clear that stricter mark, then run once on held-out years.

Common mistakes

Check yourself

  1. A rule made 90 points gross over 200 trades. One trade costs 0.15 points. Just holding made 50 points over the same days, after its cost. What is the rule's lead over holding?
    Answer

    net = gross − n × c = 90 − 200 × 0.15 = 90 − 30 = 60. lead = net − hold = 60 − 50 = 10 points ahead.

  2. Someone tried 25 versions and accepts α = 5 in 100 for the whole search. How often may luck reach the pass mark for each version?
    Answer

    chance per version = α ÷ n = 5 ÷ 25 = 0.2 in 100, which is 1 time in 500.

  3. You try 30 versions on one set of years and run the best one on held-out years. It ends behind holding. You change one setting and run the held-out years again, and now it ends ahead. Does the second run count as a held-out test?
    Answer

    No. The setting was changed after seeing the held-out years, so those years helped choose the rule and are no longer held out. Count the changed rule as version 31. A fair test of it needs years nothing has touched.

  4. You and a friend code “buy when momentum turns up, sell when it fades” and backtest it on the same prices. You get 40 trades and your friend 90. What is the most likely reason?
    Answer

    The sentence fits more than one exact rule. “Momentum” needs a number of days, and “turns up” and “fades” can each be read several ways, so you coded two different rules. A backtest on the same prices gives the same trades every time. Write both rules out exactly and count them as two versions.

  5. In the lesson's simulation the best of 60 versions led holding by 3 points. What put it ahead?
    Answer

    Luck. The market was built so that past prices say nothing about which way the next move goes, so no setting could read it, and the top of 60 tries is the luckiest of 60. It was not lower costs: it made 267 trades and 40 of the other 59 versions traded less often and paid less.

Try it yourself

Run the simulation. Every line and number in this lesson comes from one Python script (below) on the course's shared simulated market, whose random seed is fixed. Change the costs, the versions or the held-out years and see what moves.

Keep the checklist by hand. A spreadsheet is enough: one row per trade with its result before costs and its cost; one row per version you try, kept or dropped. The second sheet's row count is your n. Add holding's total over the same years at the bottom of the first.

About QuantWright

QuantWright, which publishes this course, sells backtesting software, and several of these checks are built into it. Its backtests charge commission and slippage and fill at the next bar, and every result states its data window, data source and costs. Its Honesty Score (0–100) is corrected for how many versions were tried: it counts every distinct version this account has tested on that market in the last 30 days, so trying 50 variations does not make the best one look good. Its held-out test re-runs the chosen strategy once over years kept out of sight, and in the same study it counts once. In the app, a new chat is a new study with its own held-out test, so the discipline of looking once stays yours. quantwright.com

The script needs simmarket.py, the course's simulated market, in the same folder.

The code behind every number (Python, on the course's simulated market, seed 20261109)
Loading…

Going further: measuring one version's luck

The lesson counted: 2 of 60 beat holding, and the best one's lead did not carry over to the held-out years. Statisticians also measure each version's luck on its own: how often a rule whose timing adds nothing would do at least this well over this many trades, worked out from the version's trades and how much they swing. The measure is called a p-value (Hypothesis tests, lesson B6). For the best version by total, it is about 15 in 100: luck alone does this well about one time in seven, so it never cleared the usual 5 in 100 mark, let alone the stricter 0.083 in 100 for 60 versions. None of the 60 cleared the 5 in 100 mark after costs; before costs, one did (3.5 in 100), which is one more way costs matter.

Related versions. A 120-day and a 200-day look-back, or the same look-back sold after 5 or 20 days, trade on many of the same days. They share their luck, so 60 of them act like fewer separate tries, and α ÷ n is stricter than needed for close relatives (lesson C5, Many tests at once).

For the preciseEach version's luck check: the per-trade differences di = neti − ((1 + r)daysi − 1), with r holding's average day over the same period; t = mean(d) ÷ (sd(d) ÷ √n), sd with n − 1; the p-value is the one-sided Student-t tail with n − 1 degrees of freedom. The best version by total: n = 267, t = 1.05, p = 0.148. Bonferroni mark for 60: 0.05 ÷ 60 = 0.000833. Costs: half the bid-ask spread of the day's first five-minute bar at each fill, plus 0.05% each way. Fills at the next day's open; one position at a time; trades that would sell after a period ends are dropped.

Honest notes

Where this comes from