Quant from Scratch by QuantWright

Backtesting Essentials · Lesson 5

Sampling error and confidence intervals

Every number a backtest measures comes with an error bar. How wide it is, and what a 95% confidence interval does and doesn't promise.

Know the basics? Read the lesson notes

A strategy whose true win rate we know

Here is a made-up trading strategy. Each trade wins 1R or loses 1R (R is a fixed amount of money, say €10). We built it, so we know something you never know in real life: in the long run it wins 55% of its trades. That is its true win rate, the share of trades it wins.

A backtest runs a strategy over a stretch of trades to see what it would have done. Run one: 100 trades. Then run a few more.

One strategy, 100 trades a backtest (simulated)

Top: this backtest's 100 trades, a filled square for a win. Bottom: the win rate each backtest measured, against the true 55%.

Backtest–
Wins–
Measured win rate–

The gap between the number a backtest measures and the true one is called sampling error. It comes from which trades you happened to get: your sample. Nothing about the strategy changed between runs.

How far can luck move it?

Next you'll run 1,000 backtests of this strategy, 100 trades each. Distances here are in points: one point is one percentage point of the win rate, so 55% to 58% is 3 points. Most of the 1,000 measured win rates will land within how many points of 55%?

Keep your guess in mind. On the next screen you run all 1,000.

1,000 backtests of one strategy

Same strategy, same true 55%, 1,000 separate backtests of 100 trades. Run them and watch where the measured win rates pile up.

The win rate measured by each of 1,000 backtests (simulated)

Each bar counts the backtests that measured that win rate. Dashed line: the true 55%. Shaded: the band you picked.

Backtests run0
Lowest · highest–
Inside the band–

How wide is that? The standard deviation of the 1,000 win rates, how far a typical one lands from the middle, is 5.0 points. The standard deviation of a measured number has its own name: its standard error. On a bell curve (lesson A5), about 95 in 100 results land within two standard deviations of the middle. Here two standard errors is 2 × 5 = 10 points either side: the band that held most of yours.

One backtest is all you get

In real life you don't get 1,000 backtests, and nobody tells you the true win rate. You get one. Say it's your first: 54 wins out of 100, so 54%.

You can still work out its error bar. The standard error depends on only two things: the number of trades and the win rate. Your backtest gives you both: 100 trades, and its measured 54%, standing in for the true rate you can't see. On the last screen the 1,000 backtests measured that spread: 5.0 points. A formula gets the same 5.0 from your one backtest's two numbers (the theory card, soon, shows how; you can check it there). Two standard errors either side gives a range: 54% plus or minus about 10 points.

What does your one backtest tell you about the strategy's true win rate?

54% is your best single guess, but the 1,000 backtests showed that luck moves a measured win rate by about 5 points either way. So the true rate could be anywhere from about 44% to 64%.
The measured 54%, plus or minus about 10 points (two standard errors), gives about 44% to 64%. The true 55% is inside.
One backtest tells you quite a lot, just not one exact number: the true rate is very likely within about 10 points of 54%, so from about 44% to 64%.
A range like this, worked out from one sample, is called a confidence interval.

A bar for every backtest

Now draw a bar like yours around each of the 1,000 backtests: its own measured win rate, plus or minus 1.96 of its own standard errors. 1.96 is the precise version of "about 2"; a theory card later says where it comes from. Each bar is 18 to 20 points wide; widths differ a little because each bar uses its own measured rate.

How many of the 1,000 bars will miss the true 55%, landing wholly to one side of it?

Keep your guess in mind. On the next screen the bars drop in.

The coverage machine

Each bar below is one backtest's 95% confidence interval, with a dot at the win rate it measured. Grey bars cross the dashed line at 55%: they caught the true rate. Coloured bars missed it. Drop the first 100, then the rest.

1,000 backtests, one bar each (simulated)

Each bar: a measured win rate (the dot) ± 1.96 standard errors. Dashed line: the true 55%. The picture holds the latest 100 bars; the counts below cover every bar dropped.

Bars0
Caught 55%–
Missed–

A bar misses when its backtest was unusually lucky or unlucky: its win rate landed so far from 55%, about 9 or 10 points or more, that its bar can't reach back. That is what a 95% confidence interval means: a range made by a method that catches the true value about 95 times in 100.

Wider bars, fewer misses

The 95% is a choice, called the confidence level. Change it and every bar changes width: more standard errors either side for a higher level, fewer for a lower one. Slide to 99%, then to 80%.

The same 1,000 backtests at another confidence level

Each bar: a measured win rate ± z standard errors (z from the slider). Coloured: bars that missed 55%. The picture holds the first 100 bars; the counts below cover all 1,000.

Standard errors either side–
Bar for 54 wins of 100–
Missed 55%–
Bars that include 50%–

At 99% each bar stretches 2.58 standard errors either side (about 13 points) and 12 of 1,000 missed; at 80%, 1.28 (about 6 points), and 188 missed. A bar that never misses would run from 0% to 100% and tell you nothing. 95% is the usual middle ground, a convention rather than a law.

What the 95% is about

On the machine, every bar either caught 55% or missed it. None caught it "95% of the way". In real life you see only your one bar, and never the true rate, so you can't tell which kind yours is.

A friend's backtest of 100 trades gives a 95% confidence interval for its win rate of 48% to 68%. What does the 95% tell them?

The 95% is the method's record across many backtests, like the 943 of 1,000 on the machine. Their one bar either caught the true rate or it didn't.
The middle is just the measured win rate, and the machine's dots sat all over their bars. The 95% is the method's record: about 95 in 100 bars catch the true rate.
The bar is about the true win rate, not about future trades. A new run of 100 trades has its own luck on top, so it spreads wider than this bar. The 95% is how often the method catches the true rate.
Theory

How wide the bar is: the standard error

The standard error of a measured number is how far luck typically moves it from the true value: its standard deviation across repeat backtests. It has a formula, so you never need the 1,000 backtests.

Start with an average. The bell curve lesson showed that the total of n trades spreads by σ × √n. An average is the total divided by n, so it spreads by σ × √n ÷ n = σ ÷ √n. A win rate is an average too: count a win as 1 and a loss as 0, and their average is the share of wins. One such trade has σ = √(p(1 − p)). For p = 0.55: wins are 1 and losses 0, so the average is 0.55. A win sits 0.45 from it and a loss 0.55; the squared gaps, 0.45² and 0.55², weighted by 55% and 45%, average 0.2475, which is 0.55 × 0.45. Its square root is σ.

SE of an average = σ ÷ √n
SE of a win rate = √(p(1 − p) ÷ n)
SE
the standard error: the standard deviation of the measured number across repeat backtests
σ
the standard deviation of one trade's result (σ is the Greek letter sigma)
n
the number of trades
p
the win rate as a fraction (55% = 0.55). The true one is unknown, so in practice the measured one stands in for it
Worked example: your 1,000 backtests

p = 0.55 and n = 100: 0.55 × 0.45 = 0.2475; ÷ 100 = 0.002475; √0.002475 = 0.0497. Turn a fraction into points by × 100: 0.0497 = 5.0 points. Your 1,000 measured win rates spread by 5.0 points. With 400 trades: 0.2475 ÷ 400 = 0.00062, √0.00062 = 0.0249, 2.5 points. Four times the trades, half the standard error, because of the √n.

Holds when
  • each trade's result doesn't change the next one's (the trades are independent);
  • the true win rate stays the same through the test;
  • the number was measured once, as planned, not picked as the best of many.
Breaks when

Results come in clumps (lesson C2, Volatility clustering) or the market changes under the rule (lesson C3, Regimes and decay): then the spread is wider than the formula says. Picking the best of many backtests picks the lucky ones (lesson B7, The best of many).

Try it on new numbers. A backtest won 60% of 400 trades. What is the standard error of its win rate, in points?

0.6 × 0.4 = 0.24; ÷ 400 = 0.0006; √0.0006 = 0.0245, about 2.45 points.
√(p(1 − p) ÷ n): 0.6 × 0.4 = 0.24; ÷ 400 = 0.0006; √0.0006 = 0.0245, which is 2.45 points (0.01 is one point).
Theory

The confidence interval

A confidence interval is the measured number plus or minus a few standard errors. How many sets its confidence level: the share of backtests whose interval catches the true value. The number comes from the bell: across repeat backtests a measured number piles into a bell around the true value, and on a bell 95 in 100 results land within 1.96 standard deviations of the middle (the 68–95–99.7 rule's "95 within two", made exact: 95.4 in 100 land within 2 standard deviations, 95.0 within 1.96).

estimate ± z × SE
estimate
the number you measured: a win rate, an average trade
z
standard errors either side, set by the confidence level: 1.28 for 80%, 1.64 for 90%, 1.96 for 95%, 2.58 for 99%
SE
its standard error, from the last card
Worked example: your first backtest

54 wins of 100: SE = √(0.54 × 0.46 ÷ 100) = √0.002484 = 0.0498, 5.0 points. 1.96 × 4.98 = 9.8 points. The interval: 54 − 9.8 = 44.2% to 54 + 9.8 = 63.8%. It caught 55%. Across all 1,000 backtests, 943 intervals caught it.

Holds when
  • the standard error holds (the last card's list);
  • there are enough trades for the bell: for a win rate, at least about 10 wins and 10 losses;
  • "95%" is read as the method's record, not as a chance for one interval you have already drawn.
Breaks when

With few trades, or a win rate near 0% or 100%, the bar catches less often than it says and can even run past 100% (lesson C4, Bootstrap and permutation tests, builds intervals without the bell). Keeping the best of many backtests breaks it too (lesson B7, The best of many).

The same works for an average trade. A backtest of 64 trades made +0.3R a trade on average, and one trade's result has a standard deviation of 1.6R. What is the 95% confidence interval for its true average trade?

That divides 1.6R by 64 instead of by √64. SE = 1.6R ÷ √64 = 1.6R ÷ 8 = 0.2R; 1.96 × 0.2R = 0.39R; so 0.3 − 0.39 = −0.09R to 0.3 + 0.39 = +0.69R.
That is one standard error either side, about a 68% interval. SE = 1.6R ÷ √64 = 0.2R; 1.96 × 0.2R = 0.39R; so −0.09R to +0.69R.
SE = 1.6R ÷ √64 = 1.6R ÷ 8 = 0.2R; 1.96 × 0.2R = 0.39R; so 0.3 − 0.39 = −0.09R to 0.3 + 0.39 = +0.69R.
The interval runs below zero: from 64 trades, this backtest can't rule out that the true average trade loses money.

What this means for your backtest

Three habits follow. Turn them over one at a time.

  1. Read every number with its error bar.

    A win rate of 54% over 100 trades means "somewhere from about 44% to 64%". The same goes for the average trade: its standard error is σ ÷ √n. A number given without its trade count can't be read at all.

Practice 1 of 3

"My strategy wins 70% of the time"

The situationA course seller's strategy won 70% of its trades. The footnote says the backtest had 50 trades, each winning or losing the same amount.

What is the lower end of the 95% confidence interval for its true win rate? Type a percentage.

Hint: the standard error is √(p(1 − p) ÷ n) with p = 0.7 and n = 50. Turn it into points, multiply by 1.96, and take that from 70.

0.7 × 0.3 = 0.21; ÷ 50 = 0.0042; √0.0042 = 0.065, so 6.5 points. 1.96 × 6.5 = 12.7. 70 − 12.7 = 57.3%.
0.7 × 0.3 = 0.21; ÷ 50 = 0.0042; √0.0042 = 0.065, so 6.5 points. 1.96 × 6.5 = 12.7 points either side. 70 − 12.7 = 57.3%.
So the true rate could be anywhere from about 57% to 83%. Fifty trades can't pin it closer. And if the seller tried many versions first, even that range is too kind (lesson B7).
Practice 2 of 3

A narrower bar

The situationYour own backtest of 100 trades gives a win rate with a 95% bar of about ±10 points. You want it down to about ±5 points before you trust the number.

About how many trades do you need?

The standard error has √n underneath. To halve it, √n must double, so n must grow four times: 4 × 100 = 400 trades.
The standard error has √n underneath. To halve it, √n must double, so n must grow four times: 4 × 100 = 400 trades. Twice the trades only shrinks it to about ±7 points (10 ÷ √2).
Practice 3 of 3

Two backtests

The situationYou tested two strategies, once each. Every trade wins or loses the same amount. Strategy A won 56% of 100 trades. Strategy B won 53% of 1,600 trades.

Whose 95% confidence interval for the true win rate stays wholly above 50%?

A's is wide: √(0.56 × 0.44 ÷ 100) = 5.0 points, × 1.96 = 9.7, so 46.3% to 65.7%, which includes 50%. B's: √(0.53 × 0.47 ÷ 1,600) = 1.25 points, × 1.96 = 2.45, so 50.6% to 55.4%, wholly above.
A's includes 50%: √(0.56 × 0.44 ÷ 100) = 5.0 points, × 1.96 = 9.7, so 46.3% to 65.7%. Only B's, 50.6% to 55.4%, stays above.
B's stays above: √(0.53 × 0.47 ÷ 1,600) = 1.25 points, × 1.96 = 2.45, so 50.6% to 55.4%. A's, 46.3% to 65.7%, includes 50%.
B's: √(0.53 × 0.47 ÷ 1,600) = 1.25 points, × 1.96 = 2.45, so 50.6% to 55.4%, wholly above 50%. A's: √(0.56 × 0.44 ÷ 100) = 5.0 points, × 1.96 = 9.7, so 46.3% to 65.7%, which includes 50%.
The smaller number, measured on sixteen times the trades, is pinned down more tightly. A's higher win rate may still be the better strategy; its test is just too short to say.
✓

Lesson done

Every number a backtest measures, a win rate or an average trade, would come out different on another run of trades. Its standard error says by how much, and a 95% confidence interval, the number plus or minus 1.96 standard errors, comes from a method that catches the true value about 95 times in 100. Four times the trades halves it.

Key ideas

Sampling error
the gap between a measured number and the true one, from which trades you happened to get
Standard error (SE)
how far luck typically moves a measured number: its standard deviation across repeat backtests
Confidence interval
the measured number ± z standard errors: a range worked out from one sample
Confidence level
the share of samples whose interval catches the true value (95% is usual); a record of the method, not a chance for one interval
Point
one percentage point: 55% to 58% is 3 points

Formulas

SE of an average = σ ÷ √n  ·  SE of a win rate = √(p(1 − p) ÷ n)

confidence interval = estimate ± z × SE, with z = 1.96 for 95%

Other levels: z = 1.28 for 80%, 1.64 for 90%, 2.58 for 99%. Four times the trades, half the standard error.

Lesson notes

Every number a backtest measures, such as a win rate (the share of trades won), would come out different on another run of trades. A 95% confidence interval is a range around the measured number, made by a method that catches the true value about 95 times in 100. With a win rate that stays about the same, four times the trades halves its width.

The same strategy gives different answers

The lesson's made-up strategy wins or loses 1R per trade, where R is a fixed amount of money (say €10). We built it, so we know what you never know in real life: in the long run it wins 55% of its trades, its true win rate. A backtest runs it over a stretch of trades, here 100. In the lesson's simulation, the first eight backtests measured 54, 60, 63, 49, 62, 56, 46 and 50%. The gap between a measured number and the true one is sampling error, from which trades you happened to get (your sample).

How far luck reaches

Distances are in points: one point is one percentage point of the win rate, so 55% to 58% is 3 points. Across 1,000 backtests the win rates spread from 40% to 71%. 965 landed within 10 points of 55%, 361 within 2, and all 1,000 within 25 (the furthest, 71%, was 16 away). The standard deviation of the 1,000 win rates, how far a typical one lands from the middle, is 5.02 points, about 5.0. For a measured number it is called its standard error. On a bell curve (lesson A5) about 95 in 100 results land within two standard deviations of the middle: here 2 × 5 = 10 points either side.

One backtest is all you get

In real life you get one backtest and never see the true rate. Say yours won 54 of 100, so 54%. The standard error needs only the number of trades and the win rate, with the measured 54% standing in for the true one. The formula below gives 4.98 points, close to the 5.0 measured, so the true rate is very likely within two standard errors, about 10 points, of 54%: about 44% to 64%. A range like this, worked out from one sample, is a confidence interval. 54% is the best single guess.

A bar for every backtest

Draw a bar, an interval drawn as a line, around each of the 1,000 backtests: its own win rate plus or minus 1.96 of its own standard errors. Half-widths run from 8.9 to 9.8 points, as each bar uses its own rate. 943 bars caught the true 55%; 57 missed, because their backtest was so lucky or unlucky that its win rate landed about 9 or 10 points or more from 55%. 943 is close to 95 in 100, and that is what a 95% confidence interval means: a range made by a method that catches the true value about 95 times in 100.

Wider bars, fewer misses

The 95% is a choice, the confidence level; a higher level uses more standard errors either side. At 99% (2.58 either side), 1,000 − 988 = 12 bars missed; at 80% (1.28), 1,000 − 812 = 188. A bar that never misses would run 0% to 100% and tell you nothing.

What the 95% is about

Every bar either caught 55% or missed it. In real life you see one bar and never the true rate, so you cannot tell which kind yours is. A friend's 95% interval of 48% to 68% does not mean the true rate is 58%, the middle, with 95% certainty. Nor does it cover their next 100 trades, which spread wider.

What to do with a bar

Read every number with its trade count and bar. Four times the trades halves the bar: about ±10 points over 100 trades, ±5 over 400, ±2.5 over 1,600. A strategy has an edge when it makes money on average; with equal wins and losses, no edge means 50%. The lesson's strategy has one (55%), yet 807 of the 1,000 bars included 50%, and such a bar cannot tell an edge from none.

Formula: the standard error

The total of n trades spreads by σ × √n (lesson A5), where σ is the standard deviation of one trade's result (in R), so an average, the total ÷ n, spreads by σ × √n ÷ n = σ ÷ √n. A win rate is an average too: count a win as 1 and a loss as 0, and σ is then the spread of that score. For p = 0.55, a win sits 0.45 from the average and a loss 0.55; the squared gaps, weighted by 55% and 45%, average 0.2475 = 0.55 × 0.45, so σ = √(p(1 − p)).

SE of an average = σ ÷ √n
SE of a win rate = √(p(1 − p) ÷ n)
SE
the standard error: the standard deviation of the measured number across repeat backtests
σ
the standard deviation of one trade's result (σ is the Greek letter sigma)
n
the number of trades
p
the win rate as a fraction (55% = 0.55). The true one is unknown, so in practice the measured one stands in for it
Worked example: the lesson's 1,000 backtests

p = 0.55 and n = 100: 0.55 × 0.45 = 0.2475; ÷ 100 = 0.002475; √0.002475 = 0.0497. Turn a fraction into points by × 100: 0.0497 = 5.0 points, which matches the spread of the 1,000 measured win rates. With 400 trades: 0.2475 ÷ 400 = 0.00062, √0.00062 = 0.0249, 2.5 points. Four times the trades, half the standard error, because of the √n.

Holds whenBreaks when

Results come in clumps (lesson C2, Volatility clustering) or the market changes under the rule (lesson C3, Regimes and decay): then the spread is wider than the formula says. Picking the best of many backtests picks the lucky ones (lesson B7, The best of many).

Formula: the confidence interval

A confidence interval is the measured number plus or minus z standard errors, and z sets the confidence level. The bell gives z: a measured number piles into a bell around the true value, and 95 in 100 land within 1.96 standard deviations of the middle.

estimate ± z × SE
estimate
the number you measured: a win rate, an average trade
z
standard errors either side, set by the confidence level: 1.28 for 80%, 1.64 for 90%, 1.96 for 95%, 2.58 for 99%
SE
its standard error, from the last formula
Worked example: the first backtest

54 wins of 100: SE = √(0.54 × 0.46 ÷ 100) = √0.002484 = 0.0498, 4.98 points. 1.96 × 4.98 = 9.8 points. The interval runs from 54 − 9.8 to 54 + 9.8: 44.2% to 63.8%. It caught 55%. Across all 1,000 backtests, 943 intervals caught it.

Holds whenBreaks when

With few trades, or a win rate near 0% or 100%, the bar catches less often than it says and can even run past 100% (lesson C4, Bootstrap and permutation tests, builds intervals without the bell). Keeping the best of many backtests breaks it too (lesson B7, The best of many).

One more worked example

Made-up round numbers

A backtest wins 130 of 200 trades, each winning or losing the same amount.

Win rate: 130 ÷ 200 = 0.65, so p = 65%.

Standard error: 0.65 × 0.35 = 0.2275; ÷ 200 = 0.0011375; √0.0011375 = 0.0337, which is 3.37 points.

Bar: 1.96 × 3.37 = 6.61 points. 65 − 6.61 = 58.39 and 65 + 6.61 = 71.61, so 58.4% to 71.6%.

The bar sits wholly above 50%, so this backtest can tell an edge from none.

Common mistakes

Check yourself

  1. A backtest won 40% of 100 trades. What is its 95% confidence interval for the true win rate?
    Answer

    SE = √(0.4 × 0.6 ÷ 100) = √0.0024 = 0.0490, 4.90 points. 1.96 × 4.90 = 9.6 points. The interval is 40 − 9.6 = 30.4% to 40 + 9.6 = 49.6%.

  2. A backtest of 100 trades made +0.2R a trade on average, and one trade's result has a standard deviation of 1.5R. What is the 95% interval for its true average trade, and what does it say?
    Answer

    SE of an average = σ ÷ √n = 1.5R ÷ √100 = 1.5R ÷ 10 = 0.15R. 1.96 × 0.15R = 0.294R, about 0.29R. The interval runs from 0.2 − 0.294 = −0.09R to 0.2 + 0.294 = +0.49R. It includes zero, so this backtest cannot rule out that the true average trade loses money.

  3. A win rate measured on 200 trades has a 95% bar of about ±7 points. About how many trades would make it about ±3.5?
    Answer

    The standard error has √n underneath, so halving the bar means four times the trades: 4 × 200 = 800 trades.

  4. A friend's backtest of 100 trades gives a 95% interval of 48% to 68%. Does that mean their next 100 trades will win between 48% and 68% of the time?
    Answer

    No. The 95% is the method's record: about 95 in 100 intervals made this way catch the true win rate. This one either caught it or did not. The interval is about the true win rate, not about future trades, and a new run of trades has its own luck on top, so it spreads wider than the bar.

  5. Strategy X won 58% of 100 trades; strategy Y won 54% of 1,600 trades. Every trade wins or loses the same amount. Whose 95% interval stays wholly above 50%?
    Answer

    X: SE = √(0.58 × 0.42 ÷ 100) = √0.002436 = 4.94 points; 1.96 × 4.94 = 9.7; so 48.3% to 67.7%, which includes 50%. Y: SE = √(0.54 × 0.46 ÷ 1,600) = √0.00015525 = 1.246 points; 1.96 × 1.246 = 2.44; so 51.6% to 56.4%, wholly above 50%. Only Y's. X's higher win rate may still be the better strategy; its test is just too short to say.

Try it yourself

Run the simulation. Every bar and number in this lesson comes from one Python script with a fixed random seed (below). Change the true win rate, the trades per backtest or the confidence level and see what moves.

Build a bar by hand. Flip a coin 40 times and count the heads: that is one backtest of a strategy whose true win rate is 50%. Work out its 95% interval: the share of heads, plus or minus 1.96 × √(share × (1 − share) ÷ 40). Do it five times. In a spreadsheet, =IF(RAND()<0.55,1,0) in 100 cells and their average make one backtest of this lesson's strategy; copy the row down 1,000 times and count the bars that miss 0.55.

The code behind every number (Python, seed 2026110911)
Loading…

Going further: when the simple bar breaks

The interval on the theory card, estimate ± 1.96 × SE, is the simplest one. It leans on the bell, and with few trades or a win rate near 0% or 100% the bell is a poor fit. Say a backtest won 9 of 10 trades. The simple 95% interval is 90% ± 18.6 points: 71.4% to 108.6%, a win rate no strategy can have. It also catches less often than it claims: for a true win rate of 90% and 10 trades, only 65.0 of every 100 such intervals catch it.

A better-behaved choice for win rates is the Wilson interval (Wilson, 1927). It asks which true rates would make your measured one unsurprising, so it never leaves 0% to 100%, and it leans toward 50% when trades are few. For 9 of 10 it gives 59.6% to 98.2%, and for a true 90% over 10 trades it catches 93.0 times in 100. With 100 trades or more and a win rate away from the edges, the two agree closely: for this lesson's strategy both catch 94.4 times in 100.

For the precise: the Wilson interval and exact coverageWilson: with q the measured rate, n trades and z = 1.96, centre = (q + z²/(2n)) ÷ (1 + z²/n), half-width = z × √(q(1 − q)/n + z²/(4n²)) ÷ (1 + z²/n). Exact coverage (summing the binomial chance of every win count whose interval catches the true rate), simple vs Wilson, true rate 55%: 10 trades 87.3% vs 94.9%; 30 trades 93.5% vs 93.5%; 100 trades 94.4% vs 94.4%; 1,000 trades 94.8% vs 94.8%. True rate 90%: 10 trades 65.0% vs 93.0%; 30 trades 80.9% vs 97.4%; 100 trades 93.2% vs 93.6%; 1,000 trades 95.3% vs 94.9%. Coverage jumps up and down with n because win counts are whole numbers (Brown, Cai and DasGupta, 2001).
For the precise: this lesson's numbersSeed 2026110911; 1,000 backtests × 100 trades, each a win with probability 0.55 (numpy default_rng(seed).random((1000, 100)) < 0.55). Measured win rates: mean 55.0%, standard deviation 5.02 points (dividing by 1,000), lowest 40%, highest 71%; within ±2 points of 55%: 361, within ±10: 965 (exact chance 96.6%), within ±25: 1,000. Standard error at the true rate: √(0.55 × 0.45 ÷ 100) = 4.97 points. Each bar uses its own measured rate, so half-widths at 95% run from 8.9 to 9.8 points. Bars that caught 55%, all 1,000 (first 100): 80% level 812 (80), 90% 890 (87), 95% 943 (94), 99% 988 (97); exact coverage of the simple interval at 100 trades: 80.9%, 89.1%, 94.4%, 98.8%. The first 95% bar to miss is backtest #14: 66 wins, 56.7% to 75.3%. 95% bars wholly above 50%: 192; including 50%: 807; wholly below: 1. z values: the normal distribution's two-sided 80%, 90%, 95% and 99% points, 1.2816, 1.6449, 1.9600, 2.5758.

Honest notes

Where this comes from