Ready to go further
Next: test the average trade, not the best run
A single great curve is the thing overfitting produces. Check the average result per trade after costs, and how many trades sit behind it.
What is included
- What it is, in plain words
- The formula or mechanism, with exact arithmetic
- How it hits grid, martingale and DCA bots
- A check you can run before you switch a bot on
Who it is for
- People who run or plan to run a trading bot
- Anyone reading a backtest or a bot sales page
- Readers who want the numbers, not the pitch
- Not advice, and not a signal service
Key facts
- Overfitting is tuning a bot or strategy so closely to past data that it fits that data's random noise instead of a repeatable pattern.
- The more settings and combinations you try, the better the best one looks by chance alone. With 4 settings of 10 values each there are 10,000 combinations to pick a winner from.
- Approximation: the best of n no-edge strategies measured over T years has an annualised Sharpe of about sqrt(2 ln n) / sqrt(T). For n = 1,000 and T = 5 that is 1.66 (calculated by us on 27 Oct 2026).
- A real pattern survives small changes to its settings. A fitted one usually collapses when a number moves a notch.
- The fixes are out-of-sample data, fewer settings, a look at neighbouring settings, and forward testing.
What overfitting is
Price data is part pattern and part noise. A pattern repeats. Noise does not. A strategy has settings, such as a moving average length, a grid spacing, a stop distance or an indicator threshold. If you adjust those settings until the backtest looks as good as possible, you are choosing the numbers that happened to fit this particular history, including its noise.
The strategy then looks excellent on the past and ordinary or poor on the future, because the noise it memorised does not come back.
Settings versus trades
Think of each free setting as a knob that can bend the equity curve. With enough knobs you can fit nearly any shape. The risk is highest when there are many knobs and few trades. A rule with six settings tested on 40 trades has far more room to fit noise than a rule with two settings tested on 2,000 trades. There is no magic ratio, but the direction is clear: more settings and fewer trades means less trust.
The multiple-comparisons effect
Every combination you test is another attempt to find something that looks good. Try one strategy and a great result is surprising. Try a thousand and a great result is expected. That is the multiple-comparisons effect.
n = number of strategies tried, T = years of data
This is an approximation. It comes from the usual result for the largest of n independent normal values, and it assumes the strategies are independent and have zero true edge. Real settings are correlated, so a real search behaves a little more mildly. The shape is what matters: the best value grows with n, and shrinks the more years of data you have.
| Strategies tried (n) | sqrt(2 ln n) | Best Sharpe, T = 5 years |
|---|---|---|
| 1 | 0.000 | 0.00 |
| 10 | 2.146 | 0.96 |
| 100 | 3.035 | 1.36 |
| 1,000 | 3.717 | 1.66 |
Calculated by us on 27 Oct 2026. For n = 1 the approximation gives 0, which is the average for a no-edge strategy; a single strategy can of course land above or below it. With 1,000 tries on five years of data, a Sharpe near 1.7 is what the best of pure noise looks like. A Sharpe of that size, reported with no mention of how many tries preceded it, is not evidence of anything.
A worked example: neighbouring settings
One of the best checks is to look at the settings next to the winner. If a lookback of 30 is the best and 25 and 35 are close behind, the idea is probably robust. If 30 is a spike and its neighbours are flat or negative, the spike is probably noise. Below are two made-up tests of the same kind of rule, net profit in R (units of risk per trade) over the same test period.
| Lookback length | Test A: net R | Test B: net R |
|---|---|---|
| 10 | -4 | 18 |
| 15 | 2 | 21 |
| 20 | -1 | 24 |
| 25 | 3 | 26 |
| 30 | 41 | 27 |
| 35 | 0 | 25 |
| 40 | 4 | 22 |
| 45 | -2 | 19 |
| 50 | 1 | 16 |
Both tests are made-up numbers. In Test A the setting 30 earns 41R and every neighbour sits between -4R and 4R. The optimiser would pick 30 and the report would look great. In Test B the best is 30 at 27R, and the results run smoothly from 16R to 27R. B tells you the idea is not very sensitive to the exact number. A tells you the result lives at one point.
How it hits bots
- Grid bots. Spacing, range, and number of levels are three knobs. Tune them on a ranging month and the bot looks perfect for that month. In a trend the same numbers hold a full stack of losers.
- Martingale bots. The multiplier, the step distance and the take profit can be tuned until the largest losing run in the sample disappears. It did not disappear in the market. It simply did not occur in that sample. See look-ahead bias for another way a backtest can mislead.
- EAs with many inputs. A product with twenty inputs and an "optimised for 2022 to 2024" default has had room to fit its history exactly.
- DCA bots. Choosing the buy interval and drop threshold that gave the best result on one coin's past is a fit to that coin's history.
How to check for it before you run a bot
- Hold back data. Fit on one period, then test untouched on a later period. If you peek at the later period and re-tune, it is no longer out of sample.
- Count the settings and the tries. Fewer free settings is better. Write down how many combinations were tested.
- Check the neighbours. Move every setting up and down a notch. Results should degrade gently, not collapse.
- Test other markets and periods. A real idea tends to work, at least a little, somewhere other than where it was found.
- Randomise the order. A Monte Carlo simulator shows how much of the result depends on the exact sequence of trades.
- Forward test. New data after the settings are frozen is the cleanest check. See forward testing.
What it does not tell you
A fitted strategy is not proven to have no edge. It means the backtest cannot tell you whether it has one. The strategy might still work, and you would have to find out on new data. The approximation above also describes the best of many no-edge strategies; it does not describe how much of any particular result is luck.
Keep Going
Related Tools and Guides
Keep exploring
FAQ
Frequently Asked Questions
Is curve fitting the same as overfitting?
In trading, yes. Both mean tuning a rule to the noise in past data so that it looks good on that history and does not hold on new data.
What is the difference between overfitting and optimising?
Optimising is the act of searching settings. Overfitting is the result when the search fits noise. Optimising on enough data, with few settings, and then checking out of sample can be sound. Optimising many settings on a short history usually overfits.
How many parameters is too many?
There is no fixed number. The risk depends on how many settings you tested, how many trades the test has, and how stable the result is when settings move. Two settings on thousands of trades is far safer than eight settings on a few dozen.
Does out-of-sample testing fix overfitting?
It helps a great deal, as long as the out-of-sample data is truly untouched. If you look at it, change settings, and test again, you have turned it back into fitted data.
Is the sqrt(2 ln n) formula exact?
No, it is an approximation for the best of n independent, no-edge results. Real settings are correlated and real data is not perfectly normal, so treat it as the shape of the effect, not a precise figure.
Can a bot be overfitted by its seller and not by me?
Yes. The default settings of a bought EA may have been tuned on past data before you got it. Ask what period they were tuned on and how many versions were tried.
Are the numbers on this page real?
Example numbers are made up to show how the maths works, and the page says so. A figure only counts as exact when the page says it was calculated on a stated date. Nothing you enter in a calculator leaves your browser.
Can I use this to decide whether to run a bot?
It can show whether a bot's numbers hold together and how much an account can lose. It cannot say what the market will do, and it is not advice. Use it to ask better questions, then read the terms of your own broker or exchange.
Is this financial advice?
No. It explains a testing problem in general terms. Examples are made up or exact arithmetic, and none of them describes a real bot.
Know the Risk Before You Run It
Free calculators and guides, added every week. Leave your details and we will email you when new ones go up.
Trading carries a risk of loss