Quant from Scratch by QuantWright

Introduction to Quant Finance · Lesson 10

Sample size

Why 10 trades say little: how far luck can move a win rate or an average after 10, 100 and 1,000 trades, and how many it takes to tell a winning strategy from a coin flip.

Know the basics? Read the lesson notes

Ten trades

Here is a made-up trading strategy. We built it, so we know something you never know in real life: in the long run it wins 60% of its trades. That is its true win rate, the share of trades it wins.

The trades you happen to see are a sample, and how many there are is the sample size. Take a sample of 10 trades. Then take five more, each from a separate run: the same strategy, started fresh.

One strategy, 10 trades at a time (simulated)

Top: this run's 10 trades, filled for a win. Bottom: a dot for each run's win rate; the orange dot is the run you are on now. Dashed: the true 60%.

Run–
Wins–
Win rate–

Six runs of one strategy, and the win rate came out at 80%, 80%, 40%, 30%, 70% and 40%. Nothing about the strategy changed. With 10 trades, each one moves the win rate by 10 points (a point is one percentage point: 40% to 50% is 10 points), so two or three lucky trades swing it a long way.

Keep trading

Run #1 won 8 of its first 10 trades: 80%, well above the true 60%. Let it trade on to 1,000 and watch its win rate so far.

Run #1's win rate so far (simulated)

The line is the share of trades won so far. Dashed: the true 60%. The marks along the bottom go 1, 10, 100, 1,000: each is ten times the last. At the end, three more runs appear in grey.

Trades so far0
Wins–
Win rate so far–

80% after 10 trades, 65% after 100 and 61.1% after 1,000. Nothing pulled it back: the lucky early trades are still in the count. They just weigh less. After 10 trades, one trade is worth 10 points of win rate (100 ÷ 10). After 1,000, a tenth of a point (100 ÷ 1,000 = 0.1).

This settling is called the law of large numbers: as trades add up, what a sample measures settles toward the true value.

How far can luck move it?

That was one run. Next come 10,000 separate runs of this strategy, and each one's win rate after 100 trades. Leave out the 250 luckiest runs and the 250 unluckiest, so you look at the middle 95 in every 100.

How far from the true 60% will those win rates reach?

Keep your guess in mind. On the next screen you see all 10,000 runs.

10,000 runs

Here are the 10,000 runs, piled into a hill of bars. The shaded band holds the middle 95 in 100 of them. It starts at 10 trades a run. Slide to 100, 400 and 1,000.

The win rate of 10,000 runs, by sample size (simulated)

Each bar counts the runs whose win rate landed there. Dashed: the true 60%. Shaded: the band.

The band–
Its width–

At 10 trades the band runs from 30% to 90%: 60 points wide. A strategy that truly wins 60% can show you almost anything. At 100 trades it still runs from 50% to 69%, about 10 points either side: the widest guess was the right one. At 1,000 it is 56.9% to 63.0%, 6.1 points wide.

More trades narrow the hill and its band, but slowly. From 10 trades to 1,000 is 100 times the trades, and the band got about 10 times narrower (60 points to 6.1), not 100 times.

Four times the trades

From 100 trades to 400, four times as many, the band went from 19 points wide to 9.6: about half. Luck shrinks with the square root of the number of trades. The square root of 4, written √4, is 2, because 2 × 2 = 4. So four times the trades gives half the width.

At 1,000 trades the band is 6.1 points wide. About how many trades would it take to halve that, to about 3 points?

One and a half times the trades narrows the band only by √1.5 ≈ 1.22: 6.1 ÷ 1.22 = 5.0 points. Halving takes four times the trades: 4 × 1,000 = 4,000.
Twice the trades narrows the band only by √2 ≈ 1.41: 6.1 ÷ 1.41 = 4.3 points. Halving takes four times the trades: 4 × 1,000 = 4,000.
Four times the trades, and √4 = 2: half the width. 4 × 1,000 = 4,000 trades.
Our runs stop at 1,000 trades, so 4,000 is the pattern carried on, not something measured here.

Strategy or coin?

Now picture a rule that trades on a fair coin flip. In the long run it wins 50% of its trades. With wins and losses of the same size it makes nothing on average: it has no edge, no advantage. Our strategy's true 60% is a big edge.

You get one sample and see only its win rate. How many trades must the sample have before our strategy nearly always shows a win rate that almost no coin sample reaches (only 2 or 3 in 100 get that high), so one sample could tell the two apart?

Keep your guess in mind. On the next screen you put the strategy's runs next to a coin's.

Two hills

Grey: 10,000 runs of the coin. Orange: the 10,000 runs of our 60% strategy. The shaded band is now the coin's: the middle 95 in 100 coin runs. A strategy run above it shows a win rate the coin rarely reaches. Try each sample size.

A coin and the strategy, 10,000 runs each (simulated)

Bars count runs by win rate. Shaded: the coin's band. Orange bar on top: the strategy's band.

Trades in each run
The coin's band–
The strategy's band–
Strategy runs above the coin's band–

At 10 trades the hills sit almost on top of each other: only 470 of the strategy's 10,000 runs land above the coin's band. At 100 trades the coin's band runs from 40% to 60% and the strategy's from 50% to 69%, so they overlap. More than half of the strategy's runs, 5,376, land inside the coin's band. At 400 trades the bands have just come apart, and 9,774 runs land above: about 400 was the right guess.

So even an edge as big as 60% against 50% needs a few hundred trades before one sample shows it. A smaller edge needs more.

Theory

How far luck moves an average

The law of large numbers says a sample's average settles toward the true average: the long-run average trade, as 60% was the long-run win rate. This card says how fast. The standard deviation, written σ, is how far a typical result lands from the average. Add up n trades and wins and losses partly cancel, so the total swings only √n times as far as one trade, not n times: its standard deviation is σ × √n (lesson A5). An average is that total divided by n: for 100 trades the total spreads by σ × 10, and dividing by 100 leaves σ ÷ 10. In general:

σaverage = σ ÷ √n
σaverage
the standard deviation of a sample's average: how far luck typically puts the average of n trades from the true average (textbooks call it the standard error)
σ
the standard deviation of one trade's result (σ is the Greek letter sigma)
n
the sample size: the number of trades in the sample

How to read it: averages from many samples pile into a hill around the true average, a bell curve. About 68 in 100 land within one σaverage of it. The other 32 split evenly, so luck alone puts an average more than one σaverage above the true value about 1 time in 6.

A win rate is an average too. Count each win as 1 and each loss as 0: 6 wins in 10 trades average 6 ÷ 10 = 0.6, or 60%. With a win rate anywhere near a half, a win (1) sits about 0.5 above that average and a loss (0) about 0.5 below. So for one trade counted this way, σ is about 0.5.

Worked example: your 100-trade samples

σ ≈ 0.5 and n = 100: 0.5 ÷ √100 = 0.5 ÷ 10 = 0.05, which is 5 points of win rate. Measured across your 10,000 runs: 4.9 points.

Holds when
  • each trade's result doesn't change the next one's (the trades are independent);
  • the strategy stays the same throughout, so there is one true average.
Breaks when

Results that come in clumps spread wider than this (lesson C2, Volatility clustering), and a rule that stops working partway has no single true average (lesson C3, Regimes and decay).

For the precise: one trade's σ, exactlyCounted as 1 or 0, one trade with a true win rate p (as a fraction) has σ = √(p × (1 − p)): 0.5 at 50%, 0.49 at 60% and 0.46 at 70%. So "about 0.5" is close from 30% to 70%, and too big outside that range. Lesson B5 uses the exact version.

Try it on new numbers. A trade's result can also be counted in money. R is a fixed amount like €10, so a trade that wins €20 made +2R. One trade has a standard deviation of 2R. Over a sample of 400 trades, what is the standard deviation of the average trade, in R?

σ ÷ √n = 2R ÷ √400 = 2R ÷ 20 = 0.1R.
σ ÷ √n = 2R ÷ √400 = 2R ÷ 20 = 0.1R. If you got 40, you multiplied: 2R × 20 is the spread of the total of the 400 trades, not of their average.
Theory

The band for a win rate

Win rates from many samples pile into the hill you saw, a bell curve. On a bell, about 95 results in 100 land within two standard deviations of the middle. The band holds 95 in 100, so it reaches two standard deviations of the win rate either side of the true one: 2 × 0.5 ÷ √n = 1 ÷ √n as a fraction, which is 100 ÷ √n points. That reach is called the margin of error.

band ≈ p ± 100 ÷ √n points
p
the true win rate, in percent
n
the sample size: the number of trades in the sample
100 ÷ √n
the margin of error, in points: how far either side of p the win rates of about 95 samples in 100 land (± means that far above and that far below). A rule of thumb for win rates from about 30% to 70%
Worked example: the strategy and the coin

100 trades: 100 ÷ √100 = 100 ÷ 10 = 10 points. Strategy: 60 ± 10, so 50% to 70% (measured: 50% to 69%). Coin: 50 ± 10, so 40% to 60% (measured: the same). They overlap.

400 trades: 100 ÷ √400 = 100 ÷ 20 = 5 points. Strategy: 55% to 65%. Coin: 45% to 55%. The bands just touch: that is where yours came apart.

Why there: each band reaches the margin of error either side of its own true rate, so two bands meet when margin + margin equals the gap between the true rates (5 + 5 = 10). So two bands stop overlapping once the margin of error is half the gap.

Holds when
  • the last card's conditions hold: independent trades, one true win rate;
  • the win rate is between about 30% and 70%, with at least a few dozen trades (at 10 it is only rough).
Breaks when

You know only a sample's win rate, not the true one: turning a sample into a range for the true rate is lesson B5, Sampling error and confidence intervals.

Try it on new numbers. A fair coin is flipped 2,500 times. Where does its band reach?

√2,500 = 50, and 100 ÷ 50 = 2 points. So 50 ± 2: from 48% to 52%.
±5 points is the band for 400 trades (100 ÷ 20). For 2,500: √2,500 = 50, and 100 ÷ 50 = 2 points, so from 48% to 52%.
That divides 100 by 2,500 instead of by its square root. √2,500 = 50, and 100 ÷ 50 = 2 points, so from 48% to 52%.

What this means for your backtest

A backtest runs a trading rule over past prices to see what it would have made. Its trades are one sample. Two points follow.

  1. A win rate can't be read without its trade count.

    70% over 10 trades sits inside the coin's band (20% to 80% in our runs). 70% over 1,000 trades is far above it (46.9% to 53.2%). The win rate alone is the thing to read only when wins and losses are the same size, as here. If they differ, judge the average trade.

Practice 1 of 3

"Wins 70% of its trades"

The situationA made-up seller's page says: "This strategy wins 70% of its trades." The small print says the backtest had 10 trades, each winning or losing the same amount.

Could a fair coin, with no edge at all, have produced that win rate? (√10 ≈ 3.16; any phone calculator has a √ key.)

Far above over many trades, but not over 10. The margin of error is 100 ÷ √10 = 100 ÷ 3.16 = 31.6 points, so a coin's band runs from about 18% to 82%. 70% is inside it.
100 ÷ √10 = 100 ÷ 3.16 = 31.6 points either side of 50%: from about 18% to 82%. 70% is well inside.
More flips make a coin's band narrower, so 70% gets harder for it, not easier. Over 10 flips the band is 50 ± 31.6 points, about 18% to 82%, and 70% is inside.
Our coin runs measured 20% to 80%, not 18% to 82%, because 10 flips can only land on multiples of 10 points. Of those 10,000 runs, 1,747 won 7 or more of their first 10 flips: about 1 in 6. That doesn't show the seller's strategy has no edge. It shows that 10 trades can't tell.
Practice 2 of 3

A smaller edge

The situationYou think your rule truly wins 55% of its trades, each winning or losing the same amount. A coin wins 50%, so the gap is 5 points. You want a backtest long enough that the rule's band and the coin's band stop overlapping.

How many trades is that? Two bands stop overlapping once the margin of error, 100 ÷ √n, is half the gap.

Hint: half of 5 points is 2.5. 100 ÷ √n = 2.5 means √n = 100 ÷ 2.5 = 40. Then n = 40 × 40.

Half the gap is 2.5 points. 100 ÷ √n = 2.5, so √n = 100 ÷ 2.5 = 40, and n = 40 × 40 = 1,600 trades.
Half the gap is 2.5 points. 100 ÷ √n = 2.5, so √n = 100 ÷ 2.5 = 40, and n = 40 × 40 = 1,600 trades. If you got 400, that brings the margin of error down to 5 points, the whole gap: the bands would still overlap from 50% to 55%.
Practice 3 of 3

Your average trade

The situationYour own backtest has 64 trades. They made +0.25R a trade on average, and one trade's standard deviation is 2R. Picture the rule with no edge, so its true average is 0.

How does +0.25R compare with what luck alone does to the average of 64 trades?

Sixty-four trades steady an average less than it seems. σ ÷ √n = 2R ÷ √64 = 2R ÷ 8 = 0.25R: luck alone typically puts the average 0.25R from 0, which is where this one is. That far above happens about 1 time in 6.
Luck's typical distance is σ ÷ √n = 2R ÷ √64 = 2R ÷ 8 = 0.25R. So +0.25R is one typical distance, which luck alone gets above about 1 time in 6. Two would be +0.5R.
σ ÷ √n = 2R ÷ √64 = 2R ÷ 8 = 0.25R. The result sits one σaverage above the true 0. By the rule on the theory card, luck alone puts an average that far above about 1 time in 6: everyday luck.
✓

Lesson done

A sample's win rate or average trade is the true value plus luck, and the luck shrinks only with the square root of the sample size: four times the trades, half the luck. Ten trades can show almost anything, and even an edge as big as 60% against 50% needs a few hundred before one sample can tell it from a coin.

Key ideas

Sample
the trades you happened to get, standing in for all the trades a strategy could make
Sample size (n)
the number of trades in a sample
Law of large numbers
as trades add up, what a sample measures (a win rate, an average trade) settles toward the true value
Band
the range that holds the middle 95 in 100 samples' results
Margin of error
how far either side of the true value the results of about 95 samples in 100 land; for a win rate, about 100 ÷ √n points
Point
one percentage point: 40% to 50% is 10 points

Formulas

σaverage = σ ÷ √n  ·  win-rate band ≈ p ± 100 ÷ √n points

About 68 samples in 100 land within one σaverage of the true value, about 95 within two. Four times the trades, half the luck.

Lesson notes

A sample's win rate or average trade is the true value plus luck, and the luck shrinks with the square root of the sample size: four times the trades, half the luck. Ten trades can show almost anything, and even an edge as big as 60% against 50% needs a few hundred before one sample can tell it from a coin.

A sample is not the strategy

Take a made-up strategy that, in the long run, wins 60% of its trades. That is its true win rate, the share of trades it wins. The trades you happen to see are a sample, and how many there are is the sample size.

Six separate runs of it, 10 simulated trades each, gave win rates of 80%, 80%, 40%, 30%, 70% and 40%. With 10 trades, each one moves the win rate by 10 points (a point is one percentage point), so two or three lucky trades swing it far.

The win rate settles

The first of those runs kept trading. Its win rate so far was 80% after 10 trades, 65% after 100 and 61.1% after 1,000. Nothing pulled it back: the lucky early trades stayed in the count and just weighed less. After 10 trades one trade is worth 100 ÷ 10 = 10 points of win rate; after 1,000, 100 ÷ 1,000 = 0.1 of a point. This settling is the law of large numbers: as trades add up, what a sample measures settles toward the true value.

How wide luck is

Take 10,000 separate runs of the strategy and look at each one's win rate. Leave out the 250 luckiest and the 250 unluckiest. The range that holds the rest, the middle 95 in 100, is the band.

After 10 trades the band ran from 30% to 90%, 60 points wide. After 100 trades: 50% to 69% (19 points). After 400: 55.2% to 64.8% (9.6 points). After 1,000: 56.9% to 63.0% (6.1 points).

Four times the trades (100 to 400) only halved the band. Luck shrinks with the square root of the number of trades, and √4 = 2. By that pattern, halving the 1,000-trade band would take about 4,000 trades (the runs stop at 1,000, so this is not a measurement).

Strategy or coin?

A rule that trades on a fair coin flip wins 50% in the long run. With wins and losses of the same size it makes nothing on average: it has no edge, no advantage. The strategy's true 60% is a big edge.

A second simulation ran the coin 10,000 times. After 100 trades the coin's band ran from 40% to 60% and the strategy's from 50% to 69%, so they overlapped: 5,376 of the strategy's runs, more than half, landed inside the coin's band. After 400 trades the bands had just come apart, and 9,774 runs landed above.

Why the square root

The standard deviation, written σ, is how far a typical result lands from the average. Add up n trades and wins and losses partly cancel, so the total swings only √n times as far as one trade: its standard deviation is σ × √n (lesson A5). An average is that total divided by n, which leaves σ ÷ √n, written σaverage.

Averages from many samples pile into a bell curve around the true average, which is the long-run average trade. About 68 in 100 land within one σaverage of it. The other 32 split evenly, so luck alone puts an average more than one σaverage above the true value about 1 time in 6.

A win rate is an average too: count a win as 1 and a loss as 0. With a win rate anywhere near a half, each sits about 0.5 from that average, so σ for one trade is about 0.5. On a bell curve about 95 results in 100 land within two standard deviations of the middle. The band holds 95 in 100, so it reaches two standard deviations either side of the true win rate: 2 × 0.5 ÷ √n = 1 ÷ √n, which is 100 ÷ √n points. That reach is the margin of error.

What it means for a backtest

A backtest runs a trading rule over past prices to see what it would have made; its trades are one sample. So a win rate can't be read without its trade count. And a win rate alone only tells you something when wins and losses are the same size; otherwise judge the average trade. For example: 70% over 10 trades sits inside the coin's band (20% to 80% in these runs), while 70% over 1,000 trades is far above it (46.9% to 53.2%). Four times the trades gives half the luck, so the smaller the edge, the longer the test it needs. And a result inside the coin's band means "can't tell", not "no edge".

Formula: how far luck moves an average

σaverage = σ ÷ √n
σaverage
the standard deviation of a sample's average: how far luck typically puts the average of n trades from the true average (textbooks call it the standard error)
σ
the standard deviation of one trade's result (σ is the Greek letter sigma)
n
the sample size: the number of trades in the sample
Worked example: samples of 100 trades

σ ≈ 0.5 and n = 100: 0.5 ÷ √100 = 0.5 ÷ 10 = 0.05, which is 5 points of win rate. Measured across the 10,000 runs: 4.9 points.

Holds whenBreaks when

Results that come in clumps spread wider than this (lesson C2, Volatility clustering), and a rule that stops working partway has no single true average (lesson C3, Regimes and decay).

Formula: the band for a win rate

band ≈ p ± 100 ÷ √n points
p
the true win rate, in percent
n
the sample size: the number of trades in the sample
100 ÷ √n
the margin of error, in points: how far either side of p the win rates of about 95 samples in 100 land (± means that far above and that far below). A rule of thumb for win rates from about 30% to 70%
Worked example: the strategy and the coin

100 trades: 100 ÷ √100 = 100 ÷ 10 = 10 points. The strategy: 60 ± 10, so 50% to 70% (measured: 50% to 69%). The coin: 50 ± 10, so 40% to 60% (measured: the same). They overlap.

400 trades: 100 ÷ √400 = 100 ÷ 20 = 5 points. The strategy: 55% to 65%. The coin: 45% to 55%. The bands just touch: that is where the measured ones came apart.

Why there: each band reaches the margin of error either side of its own true rate, so two bands meet when margin + margin equals the gap between the true rates (5 + 5 = 10). So two bands stop overlapping once the margin of error is half the gap.

Holds whenBreaks when

You know only a sample's win rate, not the true one: turning a sample into a range for the true rate is lesson B5, Sampling error and confidence intervals.

One more worked example

Made-up round numbers

A rule's trades have a standard deviation of 1.5R each (R is a fixed amount of money, like €10). A sample has 225 trades.

Standard deviation of the average trade: 1.5R ÷ √225 = 1.5R ÷ 15 = 0.1R.

Now a win rate: a coin flipped 900 times. √900 = 30.

Margin of error: 100 ÷ 30 = 3.3 points.

The coin's band: 50 − 3.3 = 46.7% to 50 + 3.3 = 53.3%.

Common mistakes

Check yourself

  1. A rule truly wins 55% of its trades and a coin wins 50%. How many trades before their bands stop overlapping?
    Answer

    The gap is 5 points, and half of it is 2.5. 100 ÷ √n = 2.5, so √n = 100 ÷ 2.5 = 40, and n = 40 × 40 = 1,600 trades.

  2. A backtest of 64 trades made +0.25R a trade on average; one trade's standard deviation is 2R. With no edge, how unusual is that average?
    Answer

    σ ÷ √n = 2R ÷ √64 = 2R ÷ 8 = 0.25R. The result is one σaverage above 0. Luck alone puts an average that far above the true value about 1 time in 6: everyday luck.

  3. What is the margin of error for a win rate measured over 10,000 trades?
    Answer

    √10,000 = 100, and 100 ÷ 100 = 1 point.

  4. (From lesson A5.) Each trade wins or loses 1R with no edge, so one trade's standard deviation is 1R. What is the standard deviation of the total of 100 trades, and of their average?
    Answer

    The total: σ × √n = 1R × 10 = 10R. The average: σ ÷ √n = 1R ÷ 10 = 0.1R. More trades make the total swing more and the average swing less.

Try it yourself

Run the script. Every number in this lesson comes from one Python script with fixed random seeds (below). In Part B, change the coin's 0.5 to 0.55, so the gap to the strategy is 5 points instead of 10, and see how many of the strategy's runs still land above its band after 1,000 trades.

By hand, in a spreadsheet. Put =IF(RAND()<0.6,1,0) in A1 and copy it down 1,000 rows: each cell is one trade, 1 for a win. =AVERAGE(A1:A10) is the win rate of the first 10 trades, =AVERAGE(A1:A100) of the first 100, and =AVERAGE(A1:A1000) of all 1,000. Press F9 to draw a fresh run and watch which of the three numbers jumps most. Then compare each with =1/SQRT(10), =1/SQRT(100) and =1/SQRT(1000): the margin of error as a fraction.

The code behind every number (Python, seeds 2026110506 and 2026111011)
Loading…

Going further: 81% of 21 trades

Suppose a made-up seller shows a record of 17 wins in 21 trades: 81.0%. The script flipped a fair coin 21 times in each of 10,000 more runs. In 37 of them the coin won 17 or more: about 4 runs in 1,000. For one coin that is rare.

Now picture 1,000 people who each try one no-edge rule for 21 trades. On average about 4 of them (1,000 × 37 ÷ 10,000 = 3.7) end up with a record that good, by luck alone. Anyone shown only those records would see nothing but 81% win rates. A rare result stops being rare when many people, or many versions of a rule, each get a try. Lesson B7, The best of many, is about that.

The same record is unusual even for a strategy with a real edge. Of the 60% strategy's 10,000 runs, 379 (3.8%) won 17 or more of their first 21 trades. Its band after 21 trades ran from 38.1% to 81.0%. So an 81% record over 21 trades is mostly luck even when a true 60% is behind it.

For the precisePart A, seed 2026110506, one numpy Generator: rng.random((10000, 1000)) < 0.60 for the strategy's 10,000 runs of 1,000 trades, then rng.random((10000, 21)) < 0.5 for the 21-flip coin runs of this section. Part B, seed 2026111011, its own Generator: rng.random((10000, 1000)) < 0.5 for the coin's 10,000 runs. A band is numpy's 2.5th to 97.5th percentile of the 10,000 win rates, rounded to one decimal. With only 11 possible win rates after 10 trades, a band edge sits on a possible value: the strategy's 30% to 90% holds 9,836 of its runs, not exactly 9,500. "Above the coin's band" means a win count strictly above the coin's 97.5th percentile. The standard deviations of the 10,000 win rates after 10, 100, 400 and 1,000 trades were 15.4, 4.9, 2.4 and 1.5 points; 0.5 ÷ √n gives 15.8, 5.0, 2.5 and 1.6, and the exact √(0.6 × 0.4) ÷ √n gives 15.5, 4.9, 2.4 and 1.5. The coin's measured bands: 20% to 80%, 40% to 60%, 45% to 55% and 46.9% to 53.2%.

Honest notes

Where this comes from