Quant from Scratch by QuantWright

Backtesting Essentials · Lesson 11

Monte Carlo

A backtest shows its trades in one order. Put them in 1,000 other orders to see the rest, and what that can't tell you.

Know the basics? Read the lesson notes

One backtest, one path

A backtest runs a trading rule over past prices to see what it would have made. Here the rule is made up and a computer draws every trade. The account starts with $10,000 and risks the same fixed amount on every trade: a win adds $100, a loss takes $100 away. The rule wins 55 trades in 100. Play its 250 trades.

One backtest, 250 trades (simulated)

The account after each trade. Dashed: its $10,000 start.

Trade0 of 250
Account$10,000
Wins · losses0 · 0

137 wins and 113 losses: 137 − 113 = 24 more wins than losses, and 24 × $100 = $2,400. The account ended at $12,400.

The line is the account's path: its value after each trade, in order. Of 1,000 simulated backtests of this rule, this is the one that ended in the middle.

Its worst stretch

The peak is the highest value so far, the start included. The drawdown is how far the account is below its peak, as a percent of the peak. The deepest one in the backtest is its maximum drawdown (lesson A17).

The backtest's deepest fall (simulated)

Colour: the fall from a peak to the lowest value after it.

After trade 15 the account stood at $10,300. 21 trades later it was down to $9,000: $1,300 below the peak, and 1,300 ÷ 10,300 = 12.6%. No other fall was deeper, so 12.6% is the maximum drawdown.

Suppose the same 137 wins and 113 losses had come in another order. What would be different?

Keep your guess in mind. On the next screen you put the trades in a new order.

Same trades, new order

A reshuffle puts the backtest's own 250 trades in a new random order: the same 137 wins and 113 losses. Reshuffle five times and watch the end and the maximum drawdown.

The backtest's trades in a new order (simulated)

Grey: the backtest. Colour: the reshuffle.

OrderThe backtest
End$12,400
Maximum drawdown12.6%

Five new orders gave maximum drawdowns of 9.8%, 10.2%, 8.6%, 8.0% and 6.7%. All five ended at $12,400.

The end can't move: a fixed-amount account ends at its start plus all its trades added up, and a sum is the same in any order. The path does depend on the order, and the maximum drawdown is read off the path.

So the backtest's 12.6% is one draw. The order was luck, and any other order was as likely, as long as one trade's result doesn't change the next one's and every trade is drawn the same way.

A thousand reshuffles

Guess first: among 1,000 reshuffles, how deep is the deepest maximum drawdown? The backtest's own was 12.6%.

Reshuffles of one backtest (simulated)

Grey: the reshuffles. Dark: the one with the deepest maximum drawdown so far. Colour: the backtest.

Your guess:
Reshuffles:
Reshuffles drawn0
Every one ends at–
Deepest maximum drawdown–

Doing something random many times and counting what comes out is a Monte Carlo simulation. Each reshuffle is one run, and the picture of all the runs is a fan.

The deepest of the first 10 runs was 17.9%, of the first 100 still 17.9%, and of all 1,000 23.1%. More runs can only push the deepest one further, so it says as much about the number of runs as about the trades.

Reading the fan

Sort the 1,000 maximum drawdowns from shallowest to deepest and number their places, 1st to 1,000th. Slide to the 500th place, then on to the 950th.

Maximum drawdowns of the 1,000 reshuffles (simulated)

Each bar: the runs in a range one percent wide. Colour: the runs up to your place. Dashed: the backtest's own 12.6%.

Maximum drawdown at that place–
Runs that went deeper–

The 1,000 run from 4.5% to 23.1%. The 500th is the middle run, the median: 9.2%. 499 runs went deeper, and one more matched it exactly.

The 950th is the 95th percentile: 14.4%. 950 of the 1,000 runs were at or below it, and 50 went deeper.

For the preciseThe backtest's own 12.6% sits high on the list: 860 reshuffles were shallower and 138 deeper (2 tied). With the first 10 runs the 95th percentile is the 10th value, which is the deepest, 17.9%. With the first 100 it is the 95th value, 14.7%, and with all 1,000 the 950th, 14.4%. A percentile settles as runs are added, where the deepest run keeps moving. Many textbooks give the median of an even number of values as the average of the two middle ones; this lesson takes the 500th of 1,000, by the same rule as every other percentile.

The bootstrap: new mixes of the same trades

A reshuffle uses every trade once, so every run has 137 wins. But 137 is itself one draw: another backtest of the same rule would show a few more wins, or a few less.

A bootstrap run allows for that. It draws 250 trades from the backtest's list one at a time, each time from the whole list, so one trade can be drawn twice and another not at all. That is drawing with replacement.

1,000 bootstrap runs of the same backtest (simulated)

Grey: the bootstrap runs. Colour: the backtest.

Ends–
95th percentile of maximum drawdown–

The reshuffle fan closed at $12,400. This one stays open: its runs differ in their wins, so they ended anywhere from $7,800 to $18,200.

Their maximum drawdowns had a 95th percentile of 18.0%, against the reshuffles' 14.4%: deeper, because bootstrap runs differ in their wins as well as in their order.

Does a fan know what comes next?

Both fans came from one list. The simulation can check them against trades the list never held: for each of the 1,000 backtests it built both fans, then let the rule make 250 more trades on a fresh $10,000 account.

If a fan's 95th percentile were exactly right about those trades, their maximum drawdown would go deeper than it after about 50 of the 1,000 backtests. Guess the count for the reshuffle fans.

The next 250 trades went deeper than… (simulated, counts out of 1,000 backtests)

Your guess:

The next 250 trades went deeper than the backtest's own maximum drawdown after 482 of the 1,000 backtests, than the reshuffle fan's 95th percentile after 164, and than the bootstrap fan's after 75.

Each fan did better than the backtest's one number, and was still passed more often than 50. Here is why. The rule wins 137.5 trades in 250 on average, and its 1,000 backtests showed from 107 to 163. A reshuffle keeps its backtest's win count, and more wins tend to mean shallower falls, so a backtest with more wins than average gets a fan that is too shallow: 133 of the 164 followed one of the 478 such backtests. The other 31 followed the 522 with fewer wins: their deeper fans were still passed about 6 times in 100, so they did not make up for it. A bootstrap varies the count, but around its backtest's, not the rule's. A fan knows its list, not the rule behind it.

For the preciseOn the backtest shown, the next 250 trades had a maximum drawdown of 17.8%: deeper than its reshuffle fan's 14.4%, and just inside its bootstrap fan's 18.0%. Across the 1,000 backtests the next 250 trades went deeper than the median of the reshuffle fan after 505, about half, and deeper than the deepest of its 1,000 reshuffles after 22. Lesson B5 called the trades a backtest happened to get its sample, and its formula gives the size of a sample's luck: the standard error of a win rate over 250 trades is √(0.55 × 0.45 ÷ 250), about 3.1 points, or about 7.9 wins.

When losses come in clumps

So far every trade ignored the one before it: the trades were independent. A reshuffle leans on that when it treats the order as luck.

A second made-up rule also wins 55 trades in 100, but 4 times in 10 a trade repeats the result of the one before, so wins and losses come in clumps. Call it the streaky rule. A reshuffle breaks its clumps apart. Switch to it.

Two rules, 1,000 backtests of each (simulated)

Squares: one backtest's 250 trades in order, filled for a win; the number is each row's first trade. Bars: of 1,000 backtests, how many went deeper than the 95th percentile of their own reshuffles.

Rule:

With independent trades the real order is just one more order, so it should land beyond the 95th percentile of its own reshuffles about 50 times in 1,000. It did 52 times. For the streaky rule it did 540 times: the reshuffles had lost the clumps, and clumps of losses make deep drawdowns.

The streaky rule's next 250 trades clump too. They went deeper than the reshuffle fan's 95th percentile after 518 of the 1,000 backtests, against 164 for the independent rule.

For the preciseThe longest run of losses in a row averaged 6.4 trades in the independent rule's backtests and 10.8 in the streaky rule's. A bootstrap run also draws trades one at a time, so it breaks the clumps too: the streaky rule's next 250 trades went deeper than the bootstrap fan's 95th percentile after 385 of its 1,000 backtests, against 75 for the independent rule.

A fan can't see where its list came from

A rule has an edge when it makes money on average. Take a rule with none: it wins 50 trades in 100. Run 20 backtests of it and keep the one that ended highest, as a search keeps its best setting. Here the gain is all luck (lesson B7).

Do that 1,000 times. All 1,000 kept backtests ended above $10,000, at $12,915 on average. A reshuffle cannot change the end, so in every reshuffle fan all 1,000 runs ended at the backtest's own gain. In the median bootstrap fan, 960 of the 1,000 runs ended above the start: 96 in 100.

Guess: of every 100 kept backtests, after how many did the next 250 trades end above the start?

The best of 20 backtests of a rule with no edge (simulated, 1,000 times)

Each bar: how many in every 100 ended above the $10,000 start.

Your guess:

The next 250 trades ended above the start after 471 of the 1,000 kept backtests, 47 in 100 (ending exactly at $10,000 does not count as above).

Every run in a fan is made of its backtest's own trades. So a fan adds no evidence of an edge: a lucky list gives a fan that repeats its luck. And it is not an out-of-sample test, which runs a rule on data that played no part in tuning it (choosing its settings) or picking it (keeping it as the best of several): lesson B9.

For the preciseOn average the next 250 trades ended at $9,989. They ended below the start after 481 of the 1,000 kept backtests, and level with it after 48. Their maximum drawdown passed the reshuffle fan's 95th percentile after 653 of the 1,000. Nothing in these fans gave the search away. The trades are independent, so the real order looked like any other: it was deeper than the 95th percentile of its own reshuffles in 49 of the 1,000 kept backtests, as luck alone would have it.
Theory

The percentile of a Monte Carlo

The percentile rule says which of the sorted runs to read: the p-th percentile of N results is the one in place k, counted from the lowest.

k = p × N ÷ 100
N
the number of runs
p
the percentile wanted: 50 for the median, 95 for the 95th
k
the place of that percentile among the N results sorted from lowest to highest (round up if not whole). At most N − k results are higher
Worked example: your first 20 reshuffles

Sorted, their maximum drawdowns were 6.7, 7.7, 8.0, 8.1, 8.4, 8.6, 8.7, 8.8, 9.2, 9.2, 9.4, 9.8, 10.0, 10.2, 10.5, 10.9, 11.0, 13.5, 15.8 and 17.9 percent.

The 95th percentile: k = 95 × 20 ÷ 100 = 19, and the 19th value is 15.8%, with 20 − 19 = 1 run deeper. The median: k = 50 × 20 ÷ 100 = 10, and the 10th value is 9.2%.

One deeper run out of 20 is thin support; the 50 runs beyond the 950th of 1,000 are steadier. When k is not whole, round up: with 130 runs, 95 × 130 ÷ 100 = 123.5, so the 95th percentile is the 124th value.

Holds when

The rule itself holds for any sorted list of runs: it only sorts and counts.

Needed to read it as a statement about trades to come, and still not enough
  • the trades are independent;
  • the trades to come are like the list's: the same rule, with the same edge;
  • the list is one backtest run as planned, not tuned and not picked as the best of several.

With all three met, in this lesson's simulation, the reshuffle fan's 95th percentile was still passed after 164 of 1,000 backtests, not 50: a fan can't see its own list's luck.

Breaks when

Wins and losses clump, so a trade's result depends on the one before it (lesson C1, Time series basics): the fan comes out too shallow. The list was picked as the best of several (lesson B7, The best of many): the fan repeats its luck, as counted in this simulation. A tuned list (lesson B8, Overfitting) was not simulated here; the same reasoning applies to it.

New numbers. A Monte Carlo makes 400 runs and sorts their maximum drawdowns from shallowest to deepest. Which place holds the 95th percentile?

k = 95 × 400 ÷ 100 = 380: the 380th value from the shallowest. At most 400 − 380 = 20 runs are deeper.
Multiply the percentile by the number of runs, then divide by 100: k = 95 × 400 ÷ 100 = 380, the 380th value from the shallowest. At most 400 − 380 = 20 runs are deeper.

What this means for your backtest

  1. A backtest's maximum drawdown is one draw.

    Here the same trades in other orders gave 4.5% to 23.1%, if the order is luck.

Practice 1 of 3

"97% of the simulations made money"

The situationAn imagined sales page says: "We ran 10,000 Monte Carlo simulations of our system's backtest. 97% of them ended in profit."

What do those 10,000 runs add to the backtest?

Every run is made of the backtest's own trades, so nothing here is new data. In the lesson, backtests kept as the best of 20 from a rule with no edge had 96 bootstrap runs in 100 end above the start (in the median fan); their next 250 trades ended above it after 47 in 100.
The runs do add something: the range of paths and maximum drawdowns the same trades could have taken, if the trades are independent. What they can't add is evidence of an edge, because every run is made of the backtest's own trades.
Every run is the backtest's own trades, reordered or drawn again, so a list that ended in profit gives runs that mostly do, whatever produced the list. In the lesson, the best of 20 backtests of a rule with no edge had 96 bootstrap runs in 100 end above the start.
Practice 2 of 3

Your own fan

The situationYour own research: you make 500 bootstrap runs of your backtest's trades and sort the runs' maximum drawdowns from shallowest to deepest. You want the 90th percentile.

Which place on the sorted list holds it?

Hint: k = p × N ÷ 100, with p = 90 and N = 500.

k = 90 × 500 ÷ 100 = 450: the 450th value, with at most 500 − 450 = 50 runs deeper. That counts runs. About trades to come it is rough at best.
k = 90 × 500 ÷ 100 = 450: the 450th value from the shallowest, with at most 500 − 450 = 50 runs deeper. That counts runs. About trades to come it is rough at best.
Practice 3 of 3

Deeper than nearly every reshuffle

The situationA friend's backtest had a maximum drawdown of 30%. She reshuffles its trades 1,000 times, and only 6 runs go deeper. "So 30% was a freak," she says. "The fan shows what to expect."

What is the better reading?

With independent trades the real order is just one more order, and rarely lands beyond 994 of its own 1,000 reshuffles. In the lesson it passed its fan's 95th percentile in 52 of 1,000 independent backtests and 540 of 1,000 streaky ones. A sign, not proof. If her losses clump, the fan is too shallow.
That reading treats the order as luck. A real order deeper than 994 of its own 1,000 reshuffles is a sign, not proof, that it wasn't: in the lesson the real order passed its fan's 95th percentile in 540 of 1,000 streaky backtests, against 52 of 1,000 independent ones. If her losses clump, the fan is too shallow.
The same trades in other orders do give other drawdowns: 4.5% to 23.1% in the lesson. The question is why her real order sits beyond nearly all of them. One reading: her losses come in clumps, which a reshuffle breaks apart. Then the fan is too shallow.
✓

Lesson done

A backtest shows its trades in one order, so its maximum drawdown is one draw. A reshuffle puts the same trades in a new order: a fixed-amount account ends where it did, and its path and maximum drawdown change. A bootstrap draws the trades again with replacement, so the end changes too. A fan can describe trades to come only if the trades are independent, the trades to come are like the list's, and the list was not tuned or picked. Even then, in this simulation, the reshuffle fan's 95th percentile was passed after 164 of 1,000 backtests, where 50 would be exact. A fan is not a test on new data and adds no evidence that an edge is real.

Key ideas

Monte Carlo simulation
doing something random many times and counting what comes out; each time is one run
Reshuffle
a backtest's own trades in a new random order; a fixed-amount account's end stays, its path and maximum drawdown change
Bootstrap
a run that draws the backtest's trades again with replacement; the end changes too
Fan
the account lines of all the runs drawn together
Median
the middle of the sorted results: the 50th percentile
Percentile
the p-th percentile of N sorted results is the one in place p × N ÷ 100 (rounded up), counted from the lowest; the rule holds for any sorted list
Independent trades
trades where one result doesn't change the next one's

Formulas

k = p × N ÷ 100

end of a fixed-amount account = start + the sum of its trades, in any order

k: the place of the p-th percentile among N sorted results, rounded up; it holds for any sorted list. About trades to come, a fan's percentile can hold only for independent trades, trades to come like the list's, and a list not tuned or picked; even then, in this lesson's simulation, it was passed after 164 of 1,000 backtests, not 50.

Lesson notes

A backtest shows a rule's trades in one order, so its maximum drawdown is one number out of many it could have shown. A Monte Carlo simulation draws the same trade list again many times and counts the results. It can describe trades to come only if the trades are independent (one result doesn't change the next), the trades to come are like the list's, and the list was not tuned (its settings chosen on the same data) or picked (kept as the best of several). Even then, in the lesson's simulation, a fan's 95th percentile was passed after 164 of 1,000 backtests, where 50 would be exact. It is not a test on new data and adds no evidence that an edge (an average gain) is real. Every count below is from the lesson's simulation.

One backtest, one order

A backtest runs a trading rule over past prices to see what it would have made. The lesson's rule is made up: an account starts with $10,000 and risks the same fixed amount on every trade, so a win adds $100 and a loss takes $100 away. It wins 55 trades in 100. Of 1,000 simulated backtests of 250 trades, these notes follow one, the backtest whose end was the middle one of the 1,000 when sorted: 137 wins and 113 losses, so 137 − 113 = 24 more wins than losses, a gain of 24 × $100 = $2,400 and an end of $12,400. Its path is its value after each trade, in the order the trades came.

The peak is the highest value so far, the start included; the drawdown is how far the account is below its peak, as a percent of the peak; the deepest one is the maximum drawdown (lesson A17). This backtest stood at $10,300 after trade 15 and at $9,000 21 trades later: 1,300 ÷ 10,300 = 12.6%, its maximum drawdown.

The reshuffle

A reshuffle puts the backtest's own trades in a new random order, nothing added and nothing left out. Five reshuffles had maximum drawdowns of 9.8%, 10.2%, 8.6%, 8.0% and 6.7%, and all five ended at $12,400: a fixed-amount account ends at its start plus all its trades added up, and a sum is the same in any order. The path does depend on the order, and the maximum drawdown is read off the path. So the 12.6% is one draw: as long as one trade's result doesn't change the next one's and every trade is drawn the same way, the order was luck and any other order was as likely.

Doing something random many times and counting what comes out is a Monte Carlo simulation. Each reshuffle is one run, and the account lines of all the runs drawn together are a fan. The deepest maximum drawdown among 10 reshuffles was 17.9%, and among 1,000, those 10 included, it was 23.1%: more runs can only push the deepest one further, so it says as much about the number of runs as about the trades.

Reading the fan: the median and a percentile

Sorted from the shallowest, 4.5%, to the deepest, 23.1%, the 500th of the 1,000 reshuffles' maximum drawdowns is the median, the middle run: 9.2%, with 499 runs deeper and one more matching it exactly. The 950th is the 95th percentile: 14.4%, with 950 runs at or below it and 50 deeper. A percentile settles as runs are added (14.7% from 100 of the reshuffles, 14.4% from all 1,000), while the deepest run keeps growing.

The bootstrap

Every reshuffle has 137 wins, but 137 is itself one draw: another backtest of the same rule would show a few more wins or a few less. A bootstrap run draws 250 trades from the backtest's list one at a time, each time from the whole list (with replacement), so one trade can be drawn twice and another not at all. The 1,000 bootstrap runs ended anywhere from $7,800 to $18,200. Their maximum drawdowns had a 95th percentile of 18.0%, deeper than the reshuffles' 14.4%, because bootstrap runs differ in their wins as well as in their order.

What a fan can't know

For each of the 1,000 backtests the simulation built both fans, then let the rule make 250 more trades on a fresh $10,000 account. A 95th percentile that was exactly right about those trades would be passed after about 50 of the 1,000 backtests. The next 250 trades went deeper than the backtest's own maximum drawdown after 482, deeper than the reshuffle fan's 95th percentile after 164, and deeper than the bootstrap fan's after 75. The rule wins 137.5 trades in 250 on average, and its backtests showed from 107 to 163. A reshuffle keeps its backtest's win count, and more wins tend to mean shallower falls, so a backtest with more wins than average gets a fan that is too shallow: 133 of the 164 followed one of the 478 such backtests. The other 31 followed the 522 with fewer wins: their deeper fans were still passed about 6 times in 100, so they did not make up for it. A bootstrap varies the count, but around its backtest's, not the rule's.

Clumps. Trades are independent when one result doesn't change the next one's, and a reshuffle leans on that when it treats the order as luck. The made-up streaky rule also wins 55 trades in 100, but 4 times in 10 a trade repeats the result of the one before, so wins and losses come in clumps that a reshuffle breaks apart. With independent trades the real order is one more order, so it should land beyond the 95th percentile of its own reshuffles about 50 times in 1,000: it did 52 times. For the streaky rule it did 540 times, and its next 250 trades passed the reshuffle fan's 95th percentile after 518 of its 1,000 backtests.

A picked list. A rule has an edge when it makes money on average. From a rule with none, which wins 50 trades in 100, the simulation ran 20 backtests and kept the one that ended highest, 1,000 times over. All 1,000 kept backtests ended above the start, at $12,915 on average; a reshuffle cannot change the end, so in every reshuffle fan all 1,000 runs ended at the backtest's own gain, and in the median bootstrap fan (the middle one of the 1,000 fans, sorted by how many of their runs ended above the start) 960 of the 1,000 runs did. The next 250 trades ended above the start after 471 of the 1,000 kept backtests (ending exactly at $10,000 does not count as above), at $9,989 on average, and passed the reshuffle fan's 95th percentile after 653. Every run in a fan is made of its backtest's own trades, so a fan adds no evidence of an edge, and it is not an out-of-sample test, which runs a rule on data that played no part in tuning it (choosing its settings) or picking it (keeping it as the best of several): lesson B9.

Formula: the percentile of a Monte Carlo

The percentile rule says which of the sorted runs to read: the p-th percentile of N results is the one in place k, counted from the lowest.

k = p × N ÷ 100
N
the number of runs
p
the percentile wanted: 50 for the median, 95 for the 95th
k
the place of that percentile among the N results sorted from lowest to highest (round up if not whole). At most N − k results are higher
Worked example: 20 of the lesson's reshuffles

Sorted, their maximum drawdowns were 6.7, 7.7, 8.0, 8.1, 8.4, 8.6, 8.7, 8.8, 9.2, 9.2, 9.4, 9.8, 10.0, 10.2, 10.5, 10.9, 11.0, 13.5, 15.8 and 17.9 percent.

The 95th percentile: k = 95 × 20 ÷ 100 = 19, and the 19th value is 15.8%, with 20 − 19 = 1 run deeper. The median: k = 50 × 20 ÷ 100 = 10, and the 10th value is 9.2%.

One deeper run out of 20 is thin support; the 50 runs beyond the 950th of 1,000 are steadier. When k is not whole, round up: with 130 runs, 95 × 130 ÷ 100 = 123.5, so the 95th percentile is the 124th value.

Holds when

The rule itself holds for any sorted list of runs: it only sorts and counts.

Needed to read it as a statement about trades to come, and still not enough

With all three met, in this lesson's simulation, the reshuffle fan's 95th percentile was still passed after 164 of 1,000 backtests, not 50: a fan can't see its own list's luck.

Breaks when

Wins and losses clump, so a trade's result depends on the one before it (lesson C1, Time series basics): the fan comes out too shallow. The list was picked as the best of several (lesson B7, The best of many): the fan repeats its luck, as counted in this simulation. A tuned list (lesson B8, Overfitting) was not simulated here; the same reasoning applies to it.

One more worked example

Made-up round numbers

A fixed-amount account starts at $10,000 and makes three winning and three losing trades of $100. In the order win, loss, win, loss, win, loss its values are $10,100, $10,000, $10,100, $10,000, $10,100 and $10,000. The peak is $10,100 and the deepest fall from it is $100: 100 ÷ 10,100 = 1.0%.

Reshuffled with the wins first, the values are $10,100, $10,200, $10,300, $10,200, $10,100 and $10,000. The peak is $10,300 and the fall from it $300: 300 ÷ 10,300 = 2.9%. Both orders end at $10,000, because 3 wins and 3 losses add up to $0 in any order.

Common mistakes

Check yourself

  1. A Monte Carlo makes 2,000 runs and sorts their maximum drawdowns from shallowest to deepest. Which places hold the median and the 95th percentile, and how many runs can be deeper than the 95th percentile?
    Answer

    Median: k = 50 × 2,000 ÷ 100 = 1,000, the 1,000th value. 95th percentile: k = 95 × 2,000 ÷ 100 = 1,900, the 1,900th value, with at most 2,000 − 1,900 = 100 runs deeper.

  2. A fixed-amount account starts at $10,000 and makes five trades of $100. In the order win, win, loss, loss, win, what are its end and its maximum drawdown? And in the order loss, win, loss, win, win?
    Answer

    First order: $10,100, $10,200, $10,100, $10,000, $10,100. The end is $10,100; the peak is $10,200 and the lowest value after it $10,000, so the maximum drawdown is 200 ÷ 10,200 = 2.0%. Second order: $9,900, $10,000, $9,900, $10,000, $10,100. The end is again $10,100; the peak is $10,000 (the start counts) and the lowest value after it $9,900, so the maximum drawdown is 100 ÷ 10,000 = 1.0%. Same trades, same end, another maximum drawdown.

  3. Why do all reshuffles of a fixed-amount account's trades end at the same value, while bootstrap runs don't?
    Answer

    A reshuffle uses every trade exactly once, and the end is the start plus the sum of the trades, which is the same in any order. A bootstrap run draws the trades with replacement, so one trade can be drawn twice and another not at all: the mix of wins and losses changes, and the sum with it. In the lesson the 1,000 reshuffles all ended at $12,400 and the 1,000 bootstrap runs ended anywhere from $7,800 to $18,200.

  4. Name the three conditions needed for a fan's percentile to describe trades to come, and, for the two failures the lesson simulated (clumped trades, a picked list), what each did.
    Answer

    The trades are independent; the trades to come are like the list's (the same rule with the same edge); the list is one backtest run as planned, not tuned or picked. With all three met, the next 250 trades still passed the reshuffle fan's 95th percentile after 164 of 1,000 backtests, where 50 would be exact, because a fan can't see its own list's luck. With streaky trades it was 518. With a backtest kept as the best of 20 from a rule with no edge it was 653.

  5. (from lesson A17) An account's values after four trades are $10,000 at the start, then $10,400, $10,000, $9,880 and $10,200. What is its maximum drawdown? The drawdown is how far the account is below its peak, the highest value so far, as a percent of the peak.
    Answer

    The peak is $10,400 and the lowest value after it $9,880: 520 ÷ 10,400 = 5.0%. The drawdown is measured from the peak, not from the start.

Try it yourself

Run the script. Every number in this lesson comes from one Python script (below). It draws the three kinds of trade list, makes 1,000 reshuffles and 1,000 bootstrap runs of each of the 3,000 backtests, and counts what the next 250 trades did. Change the number of trades, the win rate, how often the streaky rule repeats, or the number of backtests the search keeps the best of, and see what moves.

By hand, in a spreadsheet. Put a list of trade results in column A, one a row, and =RAND() next to each in column B. Sort both columns by column B: that is one reshuffle. In column C keep the account (=C1+A2, starting from the account's first value), in column D the peak so far (=MAX(D1,C2)) and in column E the drawdown (=(D2-C2)/D2). The maximum drawdown is =MAX(E:E). Write it down, sort again, and repeat. With the results in a column, the value in place k is =SMALL(range, k).

The code behind every number (Python, numpy; seed 2026111100)
Loading…

Going further: percent-sized trades, blocks of trades, and the deepest run

Trades sized as a percent of the account. Lesson A18 called this a fixed fraction: every trade risks the same share of what the account holds. The end is then a product, not a sum (lesson A14). With 1% a trade the shown backtest would have ended at $10,000 × 1.01 × … × 0.99 × …, with 137 factors of 1.01 and 113 of 0.99: $12,555. A product is the same in any order too, so a reshuffle still keeps the end, and the path still changes.

Keeping the clumps. One way to keep part of the order is to reshuffle blocks of trades and not single trades. The script cuts each backtest into 25 blocks of 10 trades in a row and puts the blocks in a new order, 1,000 times. For the streaky rule, the next 250 trades went deeper than the 95th percentile of those runs after 274 of the 1,000 backtests, against 518 for single trades: closer, and still far from 50. For the independent rule the count was 210, against 164 for single trades. That is a count from this simulation, and no explanation for it is offered here. A block of 10 is one made-up choice, and other lengths would give other counts.

Even the deepest run. The next 250 trades went deeper than the deepest of their backtest's own 1,000 reshuffles after 22 of the 1,000 independent backtests, 243 of the streaky ones and 290 of those kept as the best of 20.

The middle of the fan. For the independent rule the next 250 trades went deeper than the median of the reshuffle fan after 505 of the 1,000 backtests, about half: in this simulation the middle of the fan was about right, and the tail was too shallow.

Honest notes

Where this comes from