Quant from Scratch by QuantWright

Testing Popular Ideas · Lesson 16

Capstone: test an idea of your own

Pick one idea from this course and take it through the five checks. You make each decision; the page keeps the score, then gives you a checklist to keep.

Know the basics? Read the lesson notes

Pick an idea

This lesson takes one idea through the five checks of the How we test an idea lesson, in order: write the exact rule, take off the costs, compare with just holding, count the versions, and keep some years held out. At each check you make a decision. The page keeps the score: a point for each decision right on the first try.

First, what the numbers can and can't show. The prices here are simulated: made up by a computer, not a real market. These markets were built to rise slowly on average, with past prices saying nothing about which way the next move goes; only the size of moves clusters, in calm and wild spells. So every count here shows what these rules do when there is nothing to find. None of it can show whether an idea works in a real market.

Which idea do you want to test? Every step after this one uses it.

You picked the moving-average cross: own the market while a fast average of recent prices is above a slow one.
You picked the RSI: buy after a run of falls, sell once the price has bounced.
You picked the opening-range breakout: trade the first break of the high or low of the day's first minutes.
The next card writes it out exactly.
Check 1 of 5

Write the exact rule

An exact rule is one that two people would code into the same trades. Here is your idea written that way, with three switches: settings where traders disagree. As in earlier lessons, each set of choices is a version, the same idea with different settings.

Moving-average cross, exact

A moving average of n days is the average of the last n daily closes (each day's last price). At each close, if the fast average is above the slow one, the rule owns the market from the next day's open; if not, it owns nothing, selling at the next open. The whole account goes in; it never bets on falls.

Switches: the fast average, 10, 20 or 50 days; the slow one, 100 or 200 days; a simple average, or an exponential one (recent days count more).

As in earlier lessons, the baseline version is just a starting point, not anyone's published rule: 50 and 200 days, simple.

RSI, "oversold, so buy", exact

The RSI (relative strength index) of n days runs from 0 to 100: 100 × the average daily rise ÷ (the average daily rise + the average daily fall), averaged over about the last n days and smoothed so recent days count more (the RSI lesson). At a close, if the rule owns nothing and the RSI is below the lower level, it buys at the next day's open; once the RSI closes above the exit level, it sells at the next open. The whole account goes in; it never bets on falls.

Switches: n = 2, 5 or 14 days; the lower level, 30 or 20; the exit level, 50 or the upper level (70 with 30, 80 with 20).

As in earlier lessons, the baseline version is just a starting point, not anyone's published rule: 14 days, buy below 30, sell above 70.

Opening-range breakout, exact

The session runs 09:30 to 16:00 in five-minute bars. The opening range is the highest and lowest price of its first minutes. After it, the first break above the range buys; the first break below sells short (sells first and buys back later, gaining when the price falls). A stop, a set price that ends a losing trade, waits behind the entry. One trade a day; out by 16:00.

Switches: the range, 5, 15 or 30 minutes; the entry, when a bar touches the range's edge or at the close of the first bar beyond it; the stop, at the range's other side or its middle.

As in earlier lessons, the baseline version is just a starting point, not anyone's published rule: 30 minutes, touch, stop at the other side.

When do you write this down, with the switches you will try?

A look at the chart is a look at results: settings chosen after it can be chosen to fit those prices.
Then the first result has helped choose the rule, and every change is one more version to count.
Written first, nothing you see later can move it.

Writing the plan down before seeing any data has a name in science: pre-registration. Your rule and its switches are now fixed.

Check 2 of 5

Take off the costs

Every buy and every sale pays a cost: the broker's fee plus a slightly worse price than the chart shows (the Costs lesson). Your baseline version runs on one simulated market from 2016 to 2021. The years 2022 to 2024 are set aside, unseen, until the fifth check. Results are in % a year: the steady yearly growth that gives the same total.

Which costs go in?

Fixed before any result, so no result can move it.
Tiny, but paid on every trade, and some rules trade hundreds of times a year.
Costs set after the results can be set to rescue one.

Your baseline version, 2016–2021, before and after costs (simulated)

Accounts starting at 100. Orange: after costs. Grey: before costs.

It bought 4 times in six years. Before costs it made −0.38% a year; after costs, −0.52%. With so few trades, costs took about 0.1 of a point a year (a point is 1% a year).

It bought 8 times in six years. Before costs it made 3.50% a year; after costs, 3.24%. With few trades, costs took about 0.3 of a point a year (a point is 1% a year).

It traded on 259.4 days a year. Each trading day pays 0.1% to buy and 0.1% to sell, so the account keeps 0.998 of itself; 0.998 multiplied 259 times is 0.60, so costs alone take about 40% of the account a year. Before costs it lost −9.44% a year; after costs, −46.12%.

Check 3 of 5

Compare with just holding

A result needs a yardstick. What should your baseline's result after costs be measured against?

A rule can make money and still trail a rising market it owns only part of the time, or lose money and still beat a falling one.
It is the simplest alternative, over the same days.
Versions measured against each other say nothing about the market; they come at the next check.

Your baseline version and just holding, 2016–2021, after costs (simulated)

Accounts starting at 100. Orange: the baseline version. Dotted: holding.

Just holding buys at the last close of 2015, pays 0.1% once, and never sells. On this market it made −1.63% a year from 2016 to 2021: the market fell. A rule's lead is its % a year minus holding's, in points.

Your baseline's −0.52% is 1.11 points ahead. Part of that: it owned nothing on 42% of the days, and owning less of a falling market loses less. One market is one draw: across 1,000 simulated markets, this version beat holding after costs over 2016–2021 in 224.

Your baseline's 3.24% is 4.87 points ahead. Part of that: it owned nothing on 50% of the days, and owning less of a falling market loses less; the rest is which days it happened to own. One market is one draw: across 1,000 simulated markets, this version beat holding after costs over 2016–2021 in 216.

Your baseline's −46.12% is 44.49 points behind, after costs. Across 1,000 simulated markets, this version beat holding after costs over 2016–2021 in none (before costs, in 56).

Check 4 of 5

Count the versions

Your three switches have 3, 2 and 2 choices, and every choice goes with every other. How many versions do they make?

3 × 2 × 2 = 12.
Multiply, don't add: 3 × 2 × 2 = 12.

Your 12 versions: each one's lead over holding, 2016–2021, after costs (simulated)

One bar per version, in points a year. Right of the line: ahead of holding. Black: the baseline. Solid orange: the pick.

All 12 ran on 2016–2021. The pick is the one with the largest lead after costs; a fair report calls it "the best of 12", and counts the dropped versions too.

On this market 6 of the 12 beat holding after costs. The pick, 50 and 100 days, simple, led by 5.4 points a year. Across 1,000 simulated markets, at least one of the 12 beat holding after costs over these years in 434.

On this market 9 of the 12 beat holding after costs. The pick, 14 days, below 30, exit above 50, led by 7.1 points a year. Across 1,000 simulated markets, at least one of the 12 beat holding after costs over these years in 400.

On this market none of the 12 beat holding. The pick, 5 minutes, touch, stop at the other side, trailed by 29.6 points a year: the least bad. Across 1,000 simulated markets, none of the 12 beat holding after costs over these years in any market.

Check 5 of 5

Use the held-out years once

2022 to 2024 were set aside before any run: years held out, which no version has seen. What goes to them?

Then the held-out years choose the rule, and a result on them flatters the choice.
The same problem, smaller: those years would help choose between three.
One version, chosen before the run, sent once.

On this market 2022 to 2024 rose fast: just holding made 23.01% a year.

Your pick made 11.57% a year after costs: 11.44 points behind holding.

Your pick made 2.75% a year after costs: 20.26 points behind holding.

Your pick lost 33.43% a year after costs: 56.44 points behind holding.

Before the reveal: what to compare with

One market is one draw. Next you will see, across 1,000 simulated markets, how often the best of 12 on 2016–2021 stayed best: came first among the 12 again on 2022–2024, after costs. Before you see that count, decide what goes beside it.

Both chosen before the count is seen.
A count with nothing beside it can't be read: some versions stay best for reasons unrelated to reading the market.
That is half of it: it ignores that the versions pay different costs.

A blind draw, one version taken at random, would be best in about 1,000 ÷ 12 = 83 of the 1,000 (a little more when versions tie, as in the 87 and 84 below). But the 12 don't pay the same costs: versions that trade less pay less, so they can stay best with no skill at all. So the fairer comparison is the version that trades least on 2016–2021, found in each market by counting trades before looking at any result.

Does the best stay best?

Guess how often your pick stayed best on 2022–2024, out of 1,000 simulated markets, then run them. The figure shows all three ideas; yours is marked.

Stayed best on 2022–2024, of 1,000 simulated markets, after costs

For each idea: the pick (orange for yours) and the least-trading version (grey). Dashed: a blind draw. Orange arrow: your guess.

Your pick stayed best in – of the 1,000.

Moving-average cross: the pick stayed best in 109, against 87 for a blind draw and 129 for the least-trading version. No better than simply trading least.

RSI: 143, against 84 and 79: above both. That is not skill: nothing here can tell which way the next move goes, so the gap must come from fixed differences between the versions that neither comparator removes. We did not test which; how many days each owns a market that rises on average is one candidate (104 of the 143 came in the 644 markets where holding rose over both periods). These versions trade from 0.2 to 29.6 times a year.

Opening-range breakout: 506, against 83 and 424, mostly because it traded least: in 999 of the 1,000 the pick was the least-trading version or the version that differs only in where its stop sits (the price that ends a losing trade) and trades on the same days.

For the preciseA blind draw is at place 1 with chance (versions sharing place 1) ÷ 12 in each market, summed over the 1,000: 87.0, 84.1 and 83.3, a little above 83.3 where versions tie for first (in 20 and 9 markets). The pick was the least-trading version itself in 250, 77 and 368 markets.

Your verdict

On this market your pick, the best of 12, trailed just holding after costs on the held-out years. Which verdict would you sign?

More than the test showed: it ran on one simulated market, and a small real effect could hide in a test this short. Say what was measured.
It says what was measured, on which years, after costs, and claims no more.
Staying best is a count among the 12 across many markets, not a result against holding; the last card showed what drives it here.

Whichever idea you picked, "no finding" is the honest verdict here, and in these markets it is also the truth: they were built with nothing to find. What it does not show: that the idea fails in a real market. This simulation can't speak to one.

And a pass would not have been proof. Across the 1,000 simulated markets, the pick beat holding after costs on 2022–2024 in 271 (moving-average cross), 255 (RSI) and 0 (opening range). More than half of those came in the 234 markets whose 2022–2024 fell (154 and 163): a rule that owns nothing part of the time loses less when the market falls.

Theory

The chance that at least one version clears the line

Your line is the result you would count as a pass, such as beating holding after costs. Each version you try is one more chance of a result that clears it when there is nothing to find. Such a result is a false positive, and the chance that a whole search gives at least one is its family-wise error rate. If each of N versions that share no days clears the line with chance p, all N miss it with chance (1 − p) multiplied N times, so:

P(at least one) = 1 − (1 − p)N
p
the chance that one version clears the line when there is nothing to find
N
the number of versions tried
P(at least one)
the chance that at least one of the N clears it, when the N are separate tries (one's result says nothing about another's) with the same p
Worked example: the moving-average cross

The line: beating holding after costs over 2016–2021. Each of the cross's 12 versions did that in about 212 of the 1,000 simulated markets, so p = 0.212. If the 12 were separate tries: 1 − 0.78812 = 1 − 0.057 = 0.943, about 943 in 1,000. The simulation counted 434.

Holds when
  • the versions are separate tries;
  • each clears the line with about the same chance p.
Breaks when

Versions share most of their days, as these 12 do: they tend to clear the line together or miss together, so at least one clears it less often than the formula says (434 against 943 here). Picking the best of 12 is a search like this.

A search tries 20 separate versions, each clearing its line with chance 0.05 when there is nothing to find; 0.9520 = 0.358. In about how many such searches of 100 does at least one version clear it?

Hint: all 20 miss with chance 0.9520. At least one clears it the rest of the time.

1 − 0.358 = 0.642: about 64 in 100.
All 20 miss with chance 0.9520 = 0.358, so at least one clears it with chance 1 − 0.358 = 0.642: about 64 in 100.

Your score, and a checklist to keep

Your scoreAnswer the steps above to see it.
  • The rule written before any run
  • Costs fixed in advance
  • Just holding as the yardstick
  • The versions counted
  • The held-out years used once
  • Both comparisons chosen before the reveal
  • A verdict that claims no more than the test

The same steps work on any idea. Turn them over one at a time.

  1. Write the rule and every switch before any run.

    Two people who code it must get the same trades.

Practice 1 of 2

A post about staying best

The situationAn invented post: "I tried 40 versions of a breakout rule on 1,000 simulated markets. The best stayed best on held-out years in 300, against 25 for a blind draw. That's skill."

Which question should come first?

300 is far above 25. But a blind draw ignores costs: versions that trade less pay less and can stay best with no skill.
More versions add chances, not evidence.
If the least-trading version, chosen in advance, stays best about as often, the 300 is mostly about costs, not skill.
In this lesson's opening-range breakout: 506 for the best of 12, 424 for the least-trading version, 83 for a blind draw.
Practice 2 of 2

Costs on a busy rule

The situationA version makes 5% a year before costs and makes 100 round trips a year (a buy and a sale), paying 0.1% on each. 0.998100 = 0.819.

After costs, about how much does it lose, in % a year?

Hint: the account grows by 1.05 before costs and keeps 0.819 after them. Multiply.

1.05 × 0.819 = 0.860, a loss of about 14% a year. Costs turned a gain into a loss.
Before costs the account grows by 1.05; costs leave 0.819 of it; 1.05 × 0.819 = 0.860: a loss of about 14% a year.
✓

Lesson done

You took one idea through the five checks: an exact rule written before any run, costs fixed in advance, just holding as the yardstick, every version counted, held-out years used once, and a blind draw and the least-trading version chosen before reading "stayed best". These simulated markets were built so that past prices say nothing about which way the next move goes, and whichever idea you picked, the verdict was no finding: on 2022–2024, after costs, the best of 12 trailed just holding. That does not show the idea fails in a real market. And a pass would not have been proof: with nothing to find, the daily ideas' picks beat holding on those years in about a quarter of 1,000 simulated markets, more than half of those in markets that fell over those years.

Key ideas

Pre-registration
writing the rule, its switches, the costs and the held-out years down before any run
Version, the best of N
the same idea with different settings; N multiplies the switches' choices, and a fair report names the best as the best of N
Blind draw, least-trading version
what "stayed best" is compared with, both chosen before the count: one version at random, best in about 1,000 ÷ N of 1,000 markets, and the version that trades least, which pays least in costs
False positive, family-wise error rate
a result that clears the line when there is nothing to find; the chance that a search gives at least one
No finding
a verdict that says what was measured, on which years, after costs; not proof that the idea fails

Formulas

P(at least one) = 1 − (1 − p)N, for N separate versions that each clear the line with chance p when there is nothing to find; lower when versions share their days

a blind draw is best in about 1,000 ÷ N of 1,000 markets, when no two versions tie

Lesson notes

The five checks of this course, taken in order on one idea the reader picks, with 12 versions each. The simulated markets were built to rise slowly on average, with past prices saying nothing about which way the next move goes; only the size of moves clusters. So every count shows what these rules do when there is nothing to find, and none can show whether an idea works in a real market. Whichever idea is picked, the verdict on the lesson's market is no finding.

Check 1: write the exact rule, first

An exact rule is one two people would code into the same trades. Each idea has three switches (settings where traders disagree), so 3 × 2 × 2 = 12 versions (the same idea with different settings):

Writing the rule and its switches down before any run, so that nothing seen later can move it, is called pre-registration.

Checks 2 and 3: costs, then just holding

Every buy and every sale pays 0.1% of the price, fixed in advance; a round trip keeps 0.998 of the account. Just holding buys once, at the last close before the period, paying 0.1% once, and never sells. A rule's lead is its % a year (the steady yearly growth that gives the same total) minus holding's, in points. On the lesson's simulated market from 2016 to 2021, holding made −1.63% a year: the market fell. The baseline versions (the lesson's starting points, not anyone's published rules), before and after costs:

Owning nothing on part of the days of a falling market explains part of the first two leads. One market is one draw: across 1,000 simulated markets the three baselines beat holding after costs over 2016–2021 in 224, 216 and 0.

Check 4: count the versions

The pick is the version with the largest lead after costs on 2016–2021; a fair report calls it the best of 12. On the lesson's market 6, 9 and 0 of the 12 beat holding after costs. Across 1,000 simulated markets at least one of the 12 beat holding after costs over those years in 434, 400 and 0.

Check 5: the held-out years, once

2022 to 2024 were set aside before any run, and only the pick went there, once, settings unchanged. On the lesson's market those years rose fast: holding made 23.01% a year. The picks made 11.57%, 2.75% and −33.43% a year after costs: 11.44, 20.26 and 56.44 points behind.

Stayed best, and what goes beside it

"Stayed best" means the pick came first among the 12 again on 2022–2024, after costs. Before that count, two comparisons were fixed: a blind draw, one version at random, best in about 1,000 ÷ 12 = 83 of 1,000 markets (a little more when versions tie); and the least-trading version, found in each market by counting trades on 2016–2021 before looking at any result, because versions that trade less pay less in costs. Across 1,000 simulated markets:

The verdict

On the lesson's market each pick trailed holding after costs on the held-out years, so the verdict is "no finding: the best of 12 trailed holding on 2022–2024, after costs". In these markets that is also the truth, since they were built with nothing to find; it does not show that any idea fails in a real market. A pass would not have been proof either: across 1,000 simulated markets the pick beat holding after costs on 2022–2024 in 271, 255 and 0, more than half of them (154 and 163) in the 234 markets whose 2022–2024 fell, where a rule that owns nothing part of the time loses less. A real market gives one history, so this 1,000-market count can't be run there; the held-out years and the two comparisons are what you have.

Formula: the chance that at least one version clears the line

A line is the result you would count as a pass, such as beating holding after costs. A result that clears it when there is nothing to find is a false positive; the chance that a search gives at least one is its family-wise error rate.

P(at least one) = 1 − (1 − p)N
p
the chance that one version clears the line when there is nothing to find
N
the number of versions tried
P(at least one)
the chance that at least one of the N clears it, when the N are separate tries with the same p
Worked example: the moving-average cross

The line: beating holding after costs over 2016–2021. Each of the 12 versions did so in about 212 of 1,000 simulated markets: p = 0.212. As separate tries, 1 − 0.78812 = 1 − 0.057 = 0.943, about 943 in 1,000. The simulation counted 434, because the 12 share most of their days.

Holds whenBreaks when

Versions share most of their days: they clear the line together or miss together, and at least one clears it less often than the formula says.

One more worked example

Made-up round numbers

A search has two switches with 4 and 5 choices: 4 × 5 = 20 versions. A blind draw would be best on held-out years in about 1,000 ÷ 20 = 50 of 1,000 markets. If each version, with nothing to find, clears its line with chance 0.10 and the 20 were separate tries, all 20 miss with chance 0.920 = 0.122, so at least one clears it with chance 1 − 0.122 = 0.878: about 88 searches in 100.

Common mistakes

Check yourself

  1. A rule has switches with 3, 3 and 2 choices. How many versions, and in about how many of 1,000 markets would a blind draw be best?
    Answer

    3 × 3 × 2 = 18 versions; 1,000 ÷ 18 = 55.6, about 56 of 1,000, when no two versions tie.

  2. Ten separate versions each clear a line with chance 0.1 when there is nothing to find. What is the chance at least one does? (0.910 = 0.349.)
    Answer

    1 − 0.349 = 0.651, about 65 in 100. Versions that share their days would clear it less often.

  3. On the lesson's market the moving-average and RSI baselines beat just holding over 2016–2021. Why is that weak evidence?
    Answer

    Holding fell 1.63% a year there, and both rules owned nothing on part of the days (42% and half), so they lost less of the fall. One market is one draw: across 1,000 simulated markets these versions beat holding after costs over those years in 224 and 216.

  4. Your pick beat holding after costs on its held-out years. Is that a finding that the idea works?
    Answer

    It passes the check, but it is not proof. In the lesson's simulated markets, built with nothing to find, the daily ideas' picks beat holding on 2022–2024 in 271 and 255 of 1,000, more than half of them in markets that fell over those years.

Try it yourself

The rules, written before the code. All three ideas and their 12 versions each are written out in the script's opening comment, with the periods, the costs, the pick, the least-trading version, the blind draw and what counts as a finding.

Run the script. Every number in this lesson comes from one Python script (below). It imports simmarket.py, the course's simulated market, which must sit in the same folder. Swap in an idea of your own with three switches, keep the rest, and see where its pick lands against the two comparisons.

By hand. The checklist on the last card fits on an index card. Keep a second sheet with one row per version tried, kept or dropped: its row count is your N.

Written by QuantWright from the published definitions of the moving average and the RSI, the plain definition of the opening range, and the rules in the script. No code from any indicator script or charting platform.

The code behind every number (Python, imports SimMarket, seeds 2026251600 to 2026252600)
Loading…
For the preciseThe markets: SimMarket's Market 1 generator, effect=False (no planted pattern), bridge="normal" (five-minute bars with no pull-back within the day); the daily bars are built from the five-minute bars. The lesson's market: seed 2026251600; the 1,000 simulated markets: seeds 2026251601 to 2026252600. Choosing years: from the last close of 2015 to the last close of 2021 (1,566 trading days, 6.00 calendar years); held-out: to the last close of 2024 (782 days, 3.00 years). Each period is its own backtest: every version starts owning nothing at its first close; holding buys at that close and pays 0.1% once; a position open at the last close is valued there with no sale charged. % a year = (the account's growth factor)1 ÷ years − 1. Ties in the pick and the least-trading version break by list order. Average trades a year on 2016–2021, by version in list order: moving-average cross 2.3 to 0.8 (the 50- and 200-day versions least); RSI 29.6 to 0.2; opening range 212.9 (5 minutes, touch) to 260.9. Average held-out place of the pick (of 12): 6.44, 5.46 and 1.58. The theory card's p: the average over the 12 versions of the share of markets in which each beat holding after costs on 2016–2021, 0.2119 (from 0.202 to 0.224); RSI 0.1311, with at least one in 400; opening range 0.

Honest notes

Where this comes from